2023-12-04T13:53:27.748 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T13:53:27.748 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148 --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} --name dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi --owner scheduled_dis@teuthology -- /home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148/orig.config.yaml 2023-12-04T13:53:27.792 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T13:53:27.881 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148 branch: wip-dis-testing-revert description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} email: idryomov@gmail.com first_in_suite: false job_id: '7477148' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-dis-testing-revert ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 7c730b2996d0b00d56aea5ca791f90c534a13422 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 7c730b2996d0b00d56aea5ca791f90c534a13422 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing-revert sha1: 7c730b2996d0b00d56aea5ca791f90c534a13422 owner: scheduled_dis@teuthology priority: 69 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 2667 sha1: 7c730b2996d0b00d56aea5ca791f90c534a13422 sleep_before_teardown: 0 subset: 17/99 suite: rbd suite_branch: wip-dis-testing-revert suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: 7c730b2996d0b00d56aea5ca791f90c534a13422 targets: smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCfbmNcyqeHhjNg3SpAV70mufKp07o/mtrEKQB1dLC4Q+M8Bfmnpw6lvzxhJ5bkx+i89G8/Yven4H30c2gnqfP8= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODo4dFnAcHFCBEVYKDs1fTrIEVFsW1LBi7z472ULX0gW0/1v17likGLMVd2GFpVeSPR9ljaeM34RM4QVXP6ObY= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_13:47:01 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T13:53:27.881 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa; will attempt to use it 2023-12-04T13:53:27.882 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks 2023-12-04T13:53:27.882 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T13:53:27.884 INFO:teuthology.task.internal:Checking packages... 2023-12-04T13:53:27.903 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7c730b2996d0b00d56aea5ca791f90c534a13422' 2023-12-04T13:53:27.903 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T13:53:27.904 INFO:teuthology.packaging:ref: None 2023-12-04T13:53:27.904 INFO:teuthology.packaging:tag: None 2023-12-04T13:53:27.904 INFO:teuthology.packaging:branch: wip-dis-testing-revert 2023-12-04T13:53:27.904 INFO:teuthology.packaging:sha1: 7c730b2996d0b00d56aea5ca791f90c534a13422 2023-12-04T13:53:27.904 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-dis-testing-revert 2023-12-04T13:53:28.048 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7672.g7c730b29 2023-12-04T13:53:28.048 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T13:53:28.055 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T13:53:28.055 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T13:53:28.103 INFO:teuthology.task.internal:Saving configuration 2023-12-04T13:53:28.119 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T13:53:28.126 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T13:53:28.150 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 13:48:04.743361', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCfbmNcyqeHhjNg3SpAV70mufKp07o/mtrEKQB1dLC4Q+M8Bfmnpw6lvzxhJ5bkx+i89G8/Yven4H30c2gnqfP8='} 2023-12-04T13:53:28.170 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 13:48:04.745355', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODo4dFnAcHFCBEVYKDs1fTrIEVFsW1LBi7z472ULX0gW0/1v17likGLMVd2GFpVeSPR9ljaeM34RM4QVXP6ObY='} 2023-12-04T13:53:28.171 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T13:53:28.177 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-12-04T13:53:28.177 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-12-04T13:53:28.177 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T13:53:28.245 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f22e09eb9d0>, signals=[15]) 2023-12-04T13:53:28.246 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T13:53:28.252 INFO:teuthology.task.internal:Opening connections... 2023-12-04T13:53:28.252 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2023-12-04T13:53:28.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T13:53:28.342 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2023-12-04T13:53:28.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T13:53:28.427 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T13:53:28.433 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2023-12-04T13:53:28.451 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2023-12-04T13:53:28.451 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2023-12-04T13:53:28.511 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2023-12-04T13:53:28.511 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2023-12-04T13:53:28.511 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2023-12-04T13:53:28.511 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2023-12-04T13:53:28.511 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2023-12-04T13:53:28.511 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2023-12-04T13:53:28.511 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-04T13:53:28.511 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2023-12-04T13:53:28.511 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2023-12-04T13:53:28.512 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-04T13:53:28.512 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2023-12-04T13:53:28.512 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T13:53:28.512 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-04T13:53:28.512 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T13:53:28.512 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2023-12-04T13:53:28.539 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2023-12-04T13:53:28.555 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2023-12-04T13:53:28.556 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2023-12-04T13:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2023-12-04T13:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="9" 2023-12-04T13:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2023-12-04T13:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2023-12-04T13:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="9" 2023-12-04T13:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el9" 2023-12-04T13:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-04T13:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2023-12-04T13:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:LOGO="fedora-logo-icon" 2023-12-04T13:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-04T13:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2023-12-04T13:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T13:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-04T13:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T13:53:28.613 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2023-12-04T13:53:28.639 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T13:53:28.648 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T13:53:28.665 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T13:53:28.666 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2023-12-04T13:53:28.668 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2023-12-04T13:53:28.683 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T13:53:28.690 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T13:53:28.690 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2023-12-04T13:53:28.728 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2023-12-04T13:53:28.861 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T13:53:28.916 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T13:53:28.930 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-04T13:53:28.930 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-04T13:53:28.930 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T13:53:28.931 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T13:53:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T13:53:28.947 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2023-12-04T13:53:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T13:53:28.948 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2023-12-04T13:53:29.003 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-361.el9.x86_64 2023-12-04T13:53:29.003 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.14.0-361.el9.x86_64 2023-12-04T13:53:29.003 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2023-12-04T13:53:29.008 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-361.el9.x86_64 2023-12-04T13:53:29.008 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.14.0-361.el9.x86_64 2023-12-04T13:53:29.008 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2023-12-04T13:53:31.296 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.4 MB 00:00 2023-12-04T13:53:31.454 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 8.9 MB/s | 6.4 MB 00:00 2023-12-04T13:53:32.621 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2023-12-04T13:53:33.079 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2023-12-04T13:53:34.062 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 7.9 MB 00:01 2023-12-04T13:53:34.586 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 7.9 MB 00:01 2023-12-04T13:53:37.046 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 18 MB 00:01 2023-12-04T13:53:39.146 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 5.2 MB/s | 18 MB 00:03 2023-12-04T13:53:41.171 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 5.8 MB 00:00 2023-12-04T13:53:43.605 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2023-12-04T13:53:44.762 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 5.8 MB 00:02 2023-12-04T13:53:46.155 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 15 kB 00:00 2023-12-04T13:53:48.108 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2023-12-04T13:53:52.025 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 2.4 MB/s | 20 MB 00:08 2023-12-04T13:53:52.981 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2023-12-04T13:53:55.346 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-04T13:53:55.346 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-04T13:53:55.444 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T13:53:55.457 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T13:53:55.457 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2023-12-04T13:53:55.457 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T13:53:55.458 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T13:53:55.458 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-04T13:53:55.458 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-12-04T13:53:55.458 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-04T13:53:55.458 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-04T13:53:55.458 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-04T13:53:55.458 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T13:53:55.458 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T13:53:55.458 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T13:53:55.458 INFO:teuthology.orchestra.run.smithi146.stdout:Install 4 Packages 2023-12-04T13:53:55.458 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T13:53:55.459 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 96 M 2023-12-04T13:53:55.459 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 121 M 2023-12-04T13:53:55.459 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T13:53:56.814 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2023-12-04T13:53:57.573 INFO:teuthology.orchestra.run.smithi146.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 3.0 MB/s | 5.6 MB 00:01 2023-12-04T13:53:58.590 INFO:teuthology.orchestra.run.smithi146.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 7.1 MB/s | 20 MB 00:02 2023-12-04T13:53:59.104 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-04T13:53:59.104 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-04T13:53:59.185 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T13:53:59.202 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T13:53:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2023-12-04T13:53:59.202 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T13:53:59.202 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T13:53:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-04T13:53:59.203 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-04T13:53:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-04T13:53:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-04T13:53:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-04T13:53:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T13:53:59.203 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T13:53:59.203 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T13:53:59.203 INFO:teuthology.orchestra.run.smithi176.stdout:Install 4 Packages 2023-12-04T13:53:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T13:53:59.203 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 96 M 2023-12-04T13:53:59.204 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 121 M 2023-12-04T13:53:59.204 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T13:53:59.898 INFO:teuthology.orchestra.run.smithi176.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 9.8 MB/s | 5.6 MB 00:00 2023-12-04T13:54:00.565 INFO:teuthology.orchestra.run.smithi176.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 16 MB/s | 20 MB 00:01 2023-12-04T13:54:00.773 INFO:teuthology.orchestra.run.smithi146.stdout:(3/4): kernel-modules-core-5.14.0-388.el9.x86_6 10 MB/s | 32 MB 00:03 2023-12-04T13:54:01.108 INFO:teuthology.orchestra.run.smithi146.stdout:(4/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 7.0 MB/s | 38 MB 00:05 2023-12-04T13:54:01.108 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T13:54:01.108 INFO:teuthology.orchestra.run.smithi146.stdout:Total 17 MB/s | 96 MB 00:05 2023-12-04T13:54:01.307 INFO:teuthology.orchestra.run.smithi176.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 19 MB/s | 38 MB 00:01 2023-12-04T13:54:01.958 INFO:teuthology.orchestra.run.smithi176.stdout:(4/4): kernel-modules-core-5.14.0-388.el9.x86_6 16 MB/s | 32 MB 00:02 2023-12-04T13:54:01.958 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T13:54:01.958 INFO:teuthology.orchestra.run.smithi176.stdout:Total 35 MB/s | 96 MB 00:02 2023-12-04T13:54:01.962 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T13:54:02.353 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T13:54:02.353 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T13:54:02.809 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T13:54:03.157 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T13:54:03.157 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T13:54:03.672 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T13:54:03.672 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T13:54:04.444 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T13:54:04.444 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T13:54:06.649 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T13:54:07.095 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T13:54:07.307 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-04T13:54:07.432 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-04T13:54:07.731 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-04T13:54:07.857 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-04T13:54:10.168 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-04T13:54:10.341 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-04T13:54:10.602 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-04T13:54:10.774 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-04T13:54:14.948 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-04T13:54:15.149 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-04T13:54:15.380 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-04T13:54:15.497 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-04T13:54:19.770 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-04T13:54:20.137 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-04T13:54:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-04T13:54:48.648 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-04T13:54:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-04T13:54:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-04T13:54:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-04T13:54:49.018 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-04T13:54:49.067 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-04T13:54:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-04T13:54:49.445 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-04T13:54:49.445 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T13:54:49.445 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T13:54:49.445 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-04T13:54:49.445 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-04T13:54:49.445 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-04T13:54:49.446 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-04T13:54:49.446 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T13:54:49.446 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T13:54:49.488 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-04T13:54:49.488 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-04T13:54:49.488 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-04T13:54:49.488 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-04T13:54:49.910 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-04T13:54:49.910 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T13:54:49.910 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T13:54:49.910 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-04T13:54:49.910 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-04T13:54:49.910 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-04T13:54:49.910 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-04T13:54:49.910 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T13:54:49.910 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T13:54:50.006 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T13:54:50.158 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-04T13:54:50.158 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-04T13:54:50.158 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-04T13:54:50.159 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T13:54:50.159 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T13:54:50.159 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2023-12-04T13:54:50.159 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T13:54:50.159 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T13:54:50.277 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-04T13:54:50.277 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-04T13:54:50.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2023-12-04T13:54:50.311 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2023-12-04T13:54:50.311 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2023-12-04T13:54:50.311 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2023-12-04T13:54:50.311 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 68.el9 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8264254 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2023-12-04T13:54:50.312 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2023-12-04T13:54:50.313 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T13:54:50.313 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T13:54:50.313 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T13:54:50.313 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T13:54:50.313 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2023-12-04T13:54:50.313 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T13:54:50.313 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T13:54:50.314 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T13:54:50.314 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 5.14.0-388.el9.x86_64 2023-12-04T13:54:50.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T13:54:50.381 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T13:54:50.524 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-04T13:54:50.524 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-04T13:54:50.524 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-04T13:54:50.525 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T13:54:50.525 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T13:54:50.525 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2023-12-04T13:54:50.525 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T13:54:50.525 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T13:54:50.642 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-04T13:54:50.642 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-04T13:54:50.642 DEBUG:teuthology.orchestra.run.smithi176:> sudo rpm -qi grub2-tools 2023-12-04T13:54:50.676 INFO:teuthology.orchestra.run.smithi176.stdout:Name : grub2-tools 2023-12-04T13:54:50.676 INFO:teuthology.orchestra.run.smithi176.stdout:Epoch : 1 2023-12-04T13:54:50.676 INFO:teuthology.orchestra.run.smithi176.stdout:Version : 2.06 2023-12-04T13:54:50.676 INFO:teuthology.orchestra.run.smithi176.stdout:Release : 68.el9 2023-12-04T13:54:50.676 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: x86_64 2023-12-04T13:54:50.676 INFO:teuthology.orchestra.run.smithi176.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-04T13:54:50.676 INFO:teuthology.orchestra.run.smithi176.stdout:Group : Unspecified 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:Size : 8264254 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:License : GPLv3+ 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:Packager : builder@centos.org 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:Vendor : CentOS 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:Summary : Support tools for GRUB. 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:Description : 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T13:54:50.677 INFO:teuthology.orchestra.run.smithi176.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T13:54:50.678 INFO:teuthology.orchestra.run.smithi176.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T13:54:50.678 INFO:teuthology.orchestra.run.smithi176.stdout:hardware devices. 2023-12-04T13:54:50.678 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T13:54:50.678 INFO:teuthology.orchestra.run.smithi176.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T13:54:50.679 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T13:54:50.679 INFO:teuthology.task.kernel:Updating grub on smithi176 to boot 5.14.0-388.el9.x86_64 2023-12-04T13:54:50.679 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T13:54:50.920 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2023-12-04T13:54:51.279 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2023-12-04T13:54:52.406 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-04T13:54:52.433 INFO:teuthology.orchestra.run.smithi146.stderr:done 2023-12-04T13:54:52.436 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2023-12-04T13:54:52.451 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.JMdbEsalVC 2023-12-04T13:54:52.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JMdbEsalVC 2023-12-04T13:54:52.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.JMdbEsalVC 2023-12-04T13:54:52.640 INFO:teuthology.orchestra.run.smithi176.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-04T13:54:52.665 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.JMdbEsalVC is 6KB 2023-12-04T13:54:52.677 INFO:teuthology.orchestra.run.smithi176.stderr:done 2023-12-04T13:54:52.680 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2023-12-04T13:54:52.695 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.l8NmsT42ej 2023-12-04T13:54:52.695 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.l8NmsT42ej 2023-12-04T13:54:52.715 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.JMdbEsalVC 2023-12-04T13:54:52.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T13:54:52.763 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.l8NmsT42ej 2023-12-04T13:54:52.793 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-04T13:54:52.793 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-04T13:54:52.793 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-04T13:54:52.793 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-04T13:54:52.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-04T13:54:52.900 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.l8NmsT42ej is 6KB 2023-12-04T13:54:52.940 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2023-12-04T13:54:52.950 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.l8NmsT42ej 2023-12-04T13:54:52.964 DEBUG:teuthology.orchestra.run.smithi176:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T13:54:53.028 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-04T13:54:53.028 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-04T13:54:53.028 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-04T13:54:53.028 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-04T13:54:53.030 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-04T13:54:53.171 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2023-12-04T13:55:22.970 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2023-12-04T13:55:22.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-12-04T13:55:22.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T13:55:23.174 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2023-12-04T13:55:23.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-12-04T13:55:23.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T13:55:41.538 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2023-12-04T13:55:41.539 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2023-12-04T13:55:50.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-12-04T13:55:50.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T13:55:50.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-12-04T13:55:50.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T13:55:53.602 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2023-12-04T13:56:05.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-12-04T13:56:05.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T13:56:50.589 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T13:57:02.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-12-04T13:57:02.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T13:57:02.892 DEBUG:teuthology.orchestra.run.smithi176:> true 2023-12-04T13:57:03.267 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-12-04T13:57:03.267 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-04T13:57:03.267 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2023-12-04T13:57:03.282 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-388.el9.x86_64 2023-12-04T13:57:03.282 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-04T13:57:03.282 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T13:57:03.283 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T13:57:04.284 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-12-04T13:57:04.284 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T13:57:04.361 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2023-12-04T13:57:04.397 DEBUG:teuthology.parallel:result is None 2023-12-04T13:57:05.618 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T13:57:20.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-12-04T13:57:20.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T13:57:21.048 DEBUG:teuthology.orchestra.run.smithi146:> true 2023-12-04T13:57:21.397 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-12-04T13:57:21.397 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-04T13:57:21.398 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2023-12-04T13:57:21.412 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-388.el9.x86_64 2023-12-04T13:57:21.413 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-04T13:57:21.413 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T13:57:21.413 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T13:57:22.413 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-12-04T13:57:22.414 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T13:57:22.511 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2023-12-04T13:57:22.542 DEBUG:teuthology.parallel:result is None 2023-12-04T13:57:22.542 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T13:57:22.551 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T13:57:22.551 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T13:57:22.554 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T13:57:22.571 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T13:57:22.578 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T13:57:22.585 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T13:57:22.585 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T13:57:22.611 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T13:57:22.675 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T13:57:22.681 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T13:57:22.682 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T13:57:22.684 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T13:57:22.717 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T13:57:22.724 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T13:57:22.731 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T13:57:22.738 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T13:57:22.739 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T13:57:22.746 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T13:57:22.747 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T13:57:22.776 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T13:57:22.809 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T13:57:22.818 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T13:57:22.819 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T13:57:22.846 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T13:57:22.867 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T13:57:22.947 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T13:57:22.995 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T13:57:23.078 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T13:57:23.124 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T13:57:23.124 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T13:57:23.192 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T13:57:23.237 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T13:57:23.291 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T13:57:23.368 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T13:57:23.420 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T13:57:23.420 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T13:57:23.491 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2023-12-04T13:57:23.493 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2023-12-04T13:57:23.548 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T13:57:23.587 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T13:57:23.900 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T13:57:23.908 INFO:teuthology.task.internal:Starting timer... 2023-12-04T13:57:23.908 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T13:57:23.923 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T13:57:23.932 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2023-12-04T13:57:23.989 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2023-12-04T13:57:23.990 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2023-12-04T13:57:24.043 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2023-12-04T13:57:24.045 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T13:57:24.045 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2023-12-04T13:57:24.074 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2023-12-04T13:57:24.075 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2023-12-04T13:57:24.099 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2023-12-04T13:57:24.099 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi146.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi176.front.sepia.ceph.com': 'permissive'} 2023-12-04T13:57:24.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T13:57:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T13:57:24.143 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T13:57:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T13:57:24.171 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T13:57:24.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2023-12-04T13:57:24.219 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2023-12-04T13:57:24.260 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T13:57:24.270 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-04T13:57:24.270 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T13:57:24.291 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T13:57:24.292 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi146.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T14:02:53.480 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi146.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2023-12-04T14:02:53.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-12-04T14:02:53.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T14:02:53.567 DEBUG:teuthology.orchestra.run.smithi146:> true 2023-12-04T14:02:53.636 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-12-04T14:02:53.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-12-04T14:02:53.637 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T14:02:53.722 DEBUG:teuthology.orchestra.run.smithi176:> true 2023-12-04T14:02:53.788 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2023-12-04T14:02:53.789 INFO:teuthology.run_tasks:Running task clock... 2023-12-04T14:02:53.799 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-04T14:02:53.799 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-04T14:02:53.800 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-04T14:02:53.802 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-04T14:02:53.802 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-04T14:02:53.831 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-04T14:02:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-04T14:02:53.859 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-04T14:02:53.876 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-04T14:02:53.879 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2023-12-04T14:02:53.893 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2023-12-04T14:02:53.907 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2023-12-04T14:02:53.911 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-04T14:02:53.920 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2023-12-04T14:02:53.927 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-04T14:02:53.937 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-04T14:02:53.953 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-04T14:02:53.970 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2023-12-04T14:02:53.974 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-04T14:02:53.974 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2023-12-04T14:02:53.974 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T14:02:53.974 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T14:02:53.974 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T14:02:53.974 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T14:02:53.995 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2023-12-04T14:02:53.999 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-04T14:02:53.999 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2023-12-04T14:02:53.999 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T14:02:53.999 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T14:02:54.000 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T14:02:54.000 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T14:02:54.000 INFO:teuthology.run_tasks:Running task install... 2023-12-04T14:02:54.008 DEBUG:teuthology.task.install:project ceph 2023-12-04T14:02:54.009 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'}} 2023-12-04T14:02:54.009 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'} 2023-12-04T14:02:54.009 INFO:teuthology.task.install:Using flavor: default 2023-12-04T14:02:54.016 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-04T14:02:54.017 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-12-04T14:02:54.017 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422', 'tag': None, 'wait_for_package': False} 2023-12-04T14:02:54.018 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7c730b2996d0b00d56aea5ca791f90c534a13422 2023-12-04T14:02:54.020 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422', 'tag': None, 'wait_for_package': False} 2023-12-04T14:02:54.020 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7c730b2996d0b00d56aea5ca791f90c534a13422 2023-12-04T14:02:54.162 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-dis-testing-revert/7c730b2996d0b00d56aea5ca791f90c534a13422/centos/9/flavors/default/ 2023-12-04T14:02:54.162 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7672.g7c730b29 2023-12-04T14:02:54.166 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-dis-testing-revert/7c730b2996d0b00d56aea5ca791f90c534a13422/centos/9/flavors/default/ 2023-12-04T14:02:54.166 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7672.g7c730b29 2023-12-04T14:02:54.275 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing-revert/7c730b2996d0b00d56aea5ca791f90c534a13422/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing-revert/7c730b2996d0b00d56aea5ca791f90c534a13422/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing-revert/7c730b2996d0b00d56aea5ca791f90c534a13422/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-04T14:02:54.275 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:02:54.275 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-04T14:02:54.303 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-12-04T14:02:54.303 DEBUG:teuthology.orchestra.run.smithi176:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7c730b2996d0b00d56aea5ca791f90c534a13422/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-04T14:02:54.328 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing-revert/7c730b2996d0b00d56aea5ca791f90c534a13422/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing-revert/7c730b2996d0b00d56aea5ca791f90c534a13422/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing-revert/7c730b2996d0b00d56aea5ca791f90c534a13422/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-04T14:02:54.328 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:02:54.328 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-04T14:02:54.355 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-12-04T14:02:54.355 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7c730b2996d0b00d56aea5ca791f90c534a13422/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-04T14:02:54.371 DEBUG:teuthology.orchestra.run.smithi176:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-04T14:02:54.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-04T14:02:54.451 DEBUG:teuthology.orchestra.run.smithi176:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-04T14:02:54.502 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-04T14:02:54.517 INFO:teuthology.orchestra.run.smithi176.stdout:check_obsoletes = 1 2023-12-04T14:02:54.519 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum clean all 2023-12-04T14:02:54.568 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2023-12-04T14:02:54.570 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2023-12-04T14:02:54.758 INFO:teuthology.orchestra.run.smithi176.stdout:73 files removed 2023-12-04T14:02:54.785 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw 2023-12-04T14:02:54.816 INFO:teuthology.orchestra.run.smithi146.stdout:73 files removed 2023-12-04T14:02:54.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw 2023-12-04T14:02:55.493 INFO:teuthology.orchestra.run.smithi176.stdout:ceph packages for x86_64 199 kB/s | 78 kB 00:00 2023-12-04T14:02:55.656 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 158 kB/s | 78 kB 00:00 2023-12-04T14:02:55.844 INFO:teuthology.orchestra.run.smithi176.stdout:ceph noarch packages 31 kB/s | 9.5 kB 00:00 2023-12-04T14:02:55.981 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 34 kB/s | 9.5 kB 00:00 2023-12-04T14:02:56.211 INFO:teuthology.orchestra.run.smithi176.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2023-12-04T14:02:56.457 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 4.3 kB/s | 1.9 kB 00:00 2023-12-04T14:02:56.672 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-04T14:02:56.968 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-12-04T14:02:57.997 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2023-12-04T14:02:58.468 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2023-12-04T14:02:59.013 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 7.9 MB 00:00 2023-12-04T14:02:59.300 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 7.9 MB 00:00 2023-12-04T14:03:04.860 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 4.1 MB/s | 18 MB 00:04 2023-12-04T14:03:09.285 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 5.8 MB 00:01 2023-12-04T14:03:10.619 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 15 kB 00:00 2023-12-04T14:03:12.306 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2023-12-04T14:03:16.971 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-12-04T14:03:18.356 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 18 MB 00:18 2023-12-04T14:03:19.644 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:03:19.652 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:03:19.652 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-12-04T14:03:19.652 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:03:19.652 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:03:19.652 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 8.6 M 2023-12-04T14:03:19.652 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2023-12-04T14:03:19.653 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 3.3 M 2023-12-04T14:03:19.653 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 3.0 M 2023-12-04T14:03:19.653 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-12-04T14:03:19.653 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options 2023-12-04T14:03:19.653 INFO:teuthology.orchestra.run.smithi146.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-04T14:03:19.653 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 5.3 M 2023-12-04T14:03:19.653 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 19 M 2023-12-04T14:03:19.653 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 24 k 2023-12-04T14:03:19.653 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-04T14:03:19.653 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-04T14:03:19.654 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-04T14:03:19.654 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 719 k 2023-12-04T14:03:19.654 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-04T14:03:19.654 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-04T14:03:19.654 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-04T14:03:19.654 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 476 k 2023-12-04T14:03:19.654 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-04T14:03:19.654 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 4.8 M 2023-12-04T14:03:19.654 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-04T14:03:19.654 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-04T14:03:19.655 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-04T14:03:19.655 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse 2023-12-04T14:03:19.655 INFO:teuthology.orchestra.run.smithi146.stdout: x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 45 k 2023-12-04T14:03:19.655 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 129 k 2023-12-04T14:03:19.655 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 162 k 2023-12-04T14:03:19.655 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-04T14:03:19.655 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 320 k 2023-12-04T14:03:19.655 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 299 k 2023-12-04T14:03:19.655 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 99 k 2023-12-04T14:03:19.655 INFO:teuthology.orchestra.run.smithi146.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-04T14:03:19.655 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-04T14:03:19.656 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:19.656 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:03:19.656 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:03:19.656 INFO:teuthology.orchestra.run.smithi146.stdout:Install 27 Packages 2023-12-04T14:03:19.656 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 2 Packages 2023-12-04T14:03:19.656 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:19.657 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 55 M 2023-12-04T14:03:19.657 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:03:21.172 INFO:teuthology.orchestra.run.smithi146.stdout:(1/29): ceph-radosgw-18.0.0-7672.g7c730b29.el9. 5.9 MB/s | 8.6 MB 00:01 2023-12-04T14:03:21.213 INFO:teuthology.orchestra.run.smithi146.stdout:(2/29): ceph-selinux-18.0.0-7672.g7c730b29.el9. 580 kB/s | 24 kB 00:00 2023-12-04T14:03:21.297 INFO:teuthology.orchestra.run.smithi146.stdout:(3/29): libcephfs2-18.0.0-7672.g7c730b29.el9.x8 8.4 MB/s | 719 kB 00:00 2023-12-04T14:03:21.356 INFO:teuthology.orchestra.run.smithi146.stdout:(4/29): libradosstriper1-18.0.0-7672.g7c730b29. 8.0 MB/s | 476 kB 00:00 2023-12-04T14:03:21.748 INFO:teuthology.orchestra.run.smithi146.stdout:(5/29): ceph-base-18.0.0-7672.g7c730b29.el9.x86 2.6 MB/s | 5.3 MB 00:02 2023-12-04T14:03:21.831 INFO:teuthology.orchestra.run.smithi146.stdout:(6/29): librgw2-18.0.0-7672.g7c730b29.el9.x86_6 10 MB/s | 4.8 MB 00:00 2023-12-04T14:03:21.848 INFO:teuthology.orchestra.run.smithi146.stdout:(7/29): python3-ceph-argparse-18.0.0-7672.g7c73 451 kB/s | 45 kB 00:00 2023-12-04T14:03:21.890 INFO:teuthology.orchestra.run.smithi146.stdout:(8/29): python3-ceph-common-18.0.0-7672.g7c730b 2.2 MB/s | 129 kB 00:00 2023-12-04T14:03:21.965 INFO:teuthology.orchestra.run.smithi146.stdout:(9/29): python3-rados-18.0.0-7672.g7c730b29.el9 4.2 MB/s | 320 kB 00:00 2023-12-04T14:03:21.990 INFO:teuthology.orchestra.run.smithi146.stdout:(10/29): python3-cephfs-18.0.0-7672.g7c730b29.e 1.1 MB/s | 162 kB 00:00 2023-12-04T14:03:22.024 INFO:teuthology.orchestra.run.smithi146.stdout:(11/29): python3-rbd-18.0.0-7672.g7c730b29.el9. 5.0 MB/s | 299 kB 00:00 2023-12-04T14:03:22.083 INFO:teuthology.orchestra.run.smithi146.stdout:(12/29): python3-rgw-18.0.0-7672.g7c730b29.el9. 1.1 MB/s | 99 kB 00:00 2023-12-04T14:03:22.113 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 14 MB/s | 5.8 MB 00:00 2023-12-04T14:03:22.141 INFO:teuthology.orchestra.run.smithi146.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 906 kB/s | 106 kB 00:00 2023-12-04T14:03:22.166 INFO:teuthology.orchestra.run.smithi146.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2023-12-04T14:03:22.200 INFO:teuthology.orchestra.run.smithi146.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 761 kB/s | 44 kB 00:00 2023-12-04T14:03:22.242 INFO:teuthology.orchestra.run.smithi146.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2023-12-04T14:03:22.267 INFO:teuthology.orchestra.run.smithi146.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2023-12-04T14:03:22.293 INFO:teuthology.orchestra.run.smithi146.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 830 kB/s | 41 kB 00:00 2023-12-04T14:03:22.435 INFO:teuthology.orchestra.run.smithi146.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.8 MB/s | 309 kB 00:00 2023-12-04T14:03:22.477 INFO:teuthology.orchestra.run.smithi146.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 588 kB/s | 24 kB 00:00 2023-12-04T14:03:22.519 INFO:teuthology.orchestra.run.smithi146.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2023-12-04T14:03:22.577 INFO:teuthology.orchestra.run.smithi146.stdout:(22/29): libarrow-9.0.0-9.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2023-12-04T14:03:22.603 INFO:teuthology.orchestra.run.smithi146.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 807 kB/s | 67 kB 00:00 2023-12-04T14:03:22.653 INFO:teuthology.orchestra.run.smithi146.stdout:(24/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 11 MB/s | 837 kB 00:00 2023-12-04T14:03:22.678 INFO:teuthology.orchestra.run.smithi146.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2023-12-04T14:03:22.745 INFO:teuthology.orchestra.run.smithi146.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2023-12-04T14:03:23.004 INFO:teuthology.orchestra.run.smithi146.stdout:(27/29): ceph-common-18.0.0-7672.g7c730b29.el9. 5.8 MB/s | 19 MB 00:03 2023-12-04T14:03:23.221 INFO:teuthology.orchestra.run.smithi146.stdout:(28/29): librbd1-18.0.0-7672.g7c730b29.el9.x86_ 6.4 MB/s | 3.0 MB 00:00 2023-12-04T14:03:23.554 INFO:teuthology.orchestra.run.smithi146.stdout:(29/29): librados2-18.0.0-7672.g7c730b29.el9.x8 3.8 MB/s | 3.3 MB 00:00 2023-12-04T14:03:23.558 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:03:23.558 INFO:teuthology.orchestra.run.smithi146.stdout:Total 14 MB/s | 55 MB 00:03 2023-12-04T14:03:23.558 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:03:23.594 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:03:23.594 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:03:24.059 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:03:24.059 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:03:24.189 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 15 kB 00:00 2023-12-04T14:03:24.818 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 41 MB/s | 20 MB 00:00 2023-12-04T14:03:25.401 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:03:25.481 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-04T14:03:25.520 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-04T14:03:25.584 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-04T14:03:25.957 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-04T14:03:25.991 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:18.0.0-7672.g7c730b29.el9.x86_64 5/31 2023-12-04T14:03:26.074 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:18.0.0-7672.g7c730b29.el9.x86_64 5/31 2023-12-04T14:03:26.159 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:18.0.0-7672.g7c730b29.el9.x86_64 6/31 2023-12-04T14:03:26.184 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:18.0.0-7672.g7c730b29.el9.x86_64 7/31 2023-12-04T14:03:26.246 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:18.0.0-7672.g7c730b29.el9.x86_64 7/31 2023-12-04T14:03:26.275 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-04T14:03:26.319 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:18.0.0-7672.g7c730b29.el9. 9/31 2023-12-04T14:03:26.385 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:18.0.0-7672.g7c730b29.el9.x86_64 10/31 2023-12-04T14:03:26.411 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:18.0.0-7672.g7c730b29.el9.x86_6 11/31 2023-12-04T14:03:26.477 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7672.g7c730b29.el9.x86_6 11/31 2023-12-04T14:03:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-04T14:03:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-04T14:03:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-04T14:03:26.720 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-04T14:03:26.745 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-04T14:03:26.778 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-04T14:03:27.084 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-04T14:03:27.116 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:18.0.0-7672.g7c730b29.el9.x86_64 19/31 2023-12-04T14:03:27.197 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:18.0.0-7672.g7c730b29.el9.x86_64 19/31 2023-12-04T14:03:27.229 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:18.0.0-7672.g7c730b29.el9.x86_64 20/31 2023-12-04T14:03:27.265 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-04T14:03:27.693 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-04T14:03:27.727 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:18.0.0-7672.g7c730b29.el9.x86_64 23/31 2023-12-04T14:03:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:18.0.0-7672.g7c730b29.el9.x86_64 23/31 2023-12-04T14:03:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:18.0.0-7672.g7c730b29.el9.x86_64 24/31 2023-12-04T14:03:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:18.0.0-7672.g7c730b29.el9.x8 25/31 2023-12-04T14:03:29.568 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2023-12-04T14:03:29.956 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:18.0.0-7672.g7c730b29.el9.x86_64 26/31 2023-12-04T14:03:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:18.0.0-7672.g7c730b29.el9.x86_64 26/31 2023-12-04T14:03:30.577 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:18.0.0-7672.g7c730b29.el9.x86_64 26/31 2023-12-04T14:03:30.620 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:18.0.0-7672.g7c730b29.el9.x86_64 27/31 2023-12-04T14:03:30.688 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:18.0.0-7672.g7c730b29.el9.x86_64 27/31 2023-12-04T14:03:30.688 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-04T14:03:30.688 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-04T14:03:30.688 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:30.721 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:18.0.0-7672.g7c730b29.el9.x86_64 28/31 2023-12-04T14:03:32.418 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:03:32.425 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:03:32.425 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-04T14:03:32.425 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:03:32.425 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:03:32.425 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 8.6 M 2023-12-04T14:03:32.425 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2023-12-04T14:03:32.425 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 3.3 M 2023-12-04T14:03:32.426 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1 x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 3.0 M 2023-12-04T14:03:32.426 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-04T14:03:32.426 INFO:teuthology.orchestra.run.smithi176.stdout: boost-program-options 2023-12-04T14:03:32.426 INFO:teuthology.orchestra.run.smithi176.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-04T14:03:32.426 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 5.3 M 2023-12-04T14:03:32.426 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 19 M 2023-12-04T14:03:32.426 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 24 k 2023-12-04T14:03:32.426 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-04T14:03:32.426 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-04T14:03:32.426 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-04T14:03:32.426 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2 x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 719 k 2023-12-04T14:03:32.427 INFO:teuthology.orchestra.run.smithi176.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-04T14:03:32.427 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-04T14:03:32.427 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-04T14:03:32.427 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 476 k 2023-12-04T14:03:32.427 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-04T14:03:32.427 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 4.8 M 2023-12-04T14:03:32.427 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-04T14:03:32.427 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-04T14:03:32.427 INFO:teuthology.orchestra.run.smithi176.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-04T14:03:32.427 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse 2023-12-04T14:03:32.427 INFO:teuthology.orchestra.run.smithi176.stdout: x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 45 k 2023-12-04T14:03:32.427 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 129 k 2023-12-04T14:03:32.428 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 162 k 2023-12-04T14:03:32.428 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-04T14:03:32.428 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 320 k 2023-12-04T14:03:32.428 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 299 k 2023-12-04T14:03:32.428 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 99 k 2023-12-04T14:03:32.428 INFO:teuthology.orchestra.run.smithi176.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-04T14:03:32.428 INFO:teuthology.orchestra.run.smithi176.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-04T14:03:32.428 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:03:32.428 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:03:32.428 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:03:32.428 INFO:teuthology.orchestra.run.smithi176.stdout:Install 27 Packages 2023-12-04T14:03:32.428 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 2 Packages 2023-12-04T14:03:32.429 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:03:32.429 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 55 M 2023-12-04T14:03:32.429 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:03:33.044 INFO:teuthology.orchestra.run.smithi176.stdout:(1/29): ceph-base-18.0.0-7672.g7c730b29.el9.x86 9.9 MB/s | 5.3 MB 00:00 2023-12-04T14:03:33.077 INFO:teuthology.orchestra.run.smithi176.stdout:(2/29): ceph-selinux-18.0.0-7672.g7c730b29.el9. 725 kB/s | 24 kB 00:00 2023-12-04T14:03:33.152 INFO:teuthology.orchestra.run.smithi176.stdout:(3/29): libcephfs2-18.0.0-7672.g7c730b29.el9.x8 9.4 MB/s | 719 kB 00:00 2023-12-04T14:03:33.203 INFO:teuthology.orchestra.run.smithi176.stdout:(4/29): libradosstriper1-18.0.0-7672.g7c730b29. 9.3 MB/s | 476 kB 00:00 2023-12-04T14:03:33.503 INFO:teuthology.orchestra.run.smithi176.stdout:(5/29): librgw2-18.0.0-7672.g7c730b29.el9.x86_6 16 MB/s | 4.8 MB 00:00 2023-12-04T14:03:33.545 INFO:teuthology.orchestra.run.smithi176.stdout:(6/29): python3-ceph-argparse-18.0.0-7672.g7c73 1.1 MB/s | 45 kB 00:00 2023-12-04T14:03:33.587 INFO:teuthology.orchestra.run.smithi176.stdout:(7/29): python3-ceph-common-18.0.0-7672.g7c730b 3.0 MB/s | 129 kB 00:00 2023-12-04T14:03:33.637 INFO:teuthology.orchestra.run.smithi176.stdout:(8/29): python3-cephfs-18.0.0-7672.g7c730b29.el 3.2 MB/s | 162 kB 00:00 2023-12-04T14:03:33.696 INFO:teuthology.orchestra.run.smithi176.stdout:(9/29): python3-rados-18.0.0-7672.g7c730b29.el9 5.4 MB/s | 320 kB 00:00 2023-12-04T14:03:33.812 INFO:teuthology.orchestra.run.smithi176.stdout:(10/29): python3-rbd-18.0.0-7672.g7c730b29.el9. 2.5 MB/s | 299 kB 00:00 2023-12-04T14:03:33.871 INFO:teuthology.orchestra.run.smithi176.stdout:(11/29): python3-rgw-18.0.0-7672.g7c730b29.el9. 1.7 MB/s | 99 kB 00:00 2023-12-04T14:03:33.972 INFO:teuthology.orchestra.run.smithi176.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2023-12-04T14:03:34.064 INFO:teuthology.orchestra.run.smithi176.stdout:(13/29): ceph-radosgw-18.0.0-7672.g7c730b29.el9 5.5 MB/s | 8.6 MB 00:01 2023-12-04T14:03:34.089 INFO:teuthology.orchestra.run.smithi176.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2023-12-04T14:03:34.114 INFO:teuthology.orchestra.run.smithi176.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2023-12-04T14:03:34.148 INFO:teuthology.orchestra.run.smithi176.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-12-04T14:03:34.173 INFO:teuthology.orchestra.run.smithi176.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-12-04T14:03:34.198 INFO:teuthology.orchestra.run.smithi176.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 831 kB/s | 41 kB 00:00 2023-12-04T14:03:34.615 INFO:teuthology.orchestra.run.smithi176.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 699 kB/s | 309 kB 00:00 2023-12-04T14:03:34.690 INFO:teuthology.orchestra.run.smithi176.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 325 kB/s | 24 kB 00:00 2023-12-04T14:03:34.766 INFO:teuthology.orchestra.run.smithi176.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 646 kB/s | 49 kB 00:00 2023-12-04T14:03:34.949 INFO:teuthology.orchestra.run.smithi176.stdout:(22/29): ceph-common-18.0.0-7672.g7c730b29.el9. 7.8 MB/s | 19 MB 00:02 2023-12-04T14:03:34.975 INFO:teuthology.orchestra.run.smithi176.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 323 kB/s | 67 kB 00:00 2023-12-04T14:03:35.050 INFO:teuthology.orchestra.run.smithi176.stdout:(24/29): libarrow-9.0.0-9.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2023-12-04T14:03:35.100 INFO:teuthology.orchestra.run.smithi176.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2023-12-04T14:03:35.184 INFO:teuthology.orchestra.run.smithi176.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2023-12-04T14:03:35.426 INFO:teuthology.orchestra.run.smithi176.stdout:(27/29): librados2-18.0.0-7672.g7c730b29.el9.x8 10 MB/s | 3.3 MB 00:00 2023-12-04T14:03:35.501 INFO:teuthology.orchestra.run.smithi176.stdout:(28/29): librbd1-18.0.0-7672.g7c730b29.el9.x86_ 9.6 MB/s | 3.0 MB 00:00 2023-12-04T14:03:35.534 INFO:teuthology.orchestra.run.smithi176.stdout:(29/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.4 MB/s | 837 kB 00:00 2023-12-04T14:03:35.540 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:03:35.540 INFO:teuthology.orchestra.run.smithi176.stdout:Total 18 MB/s | 55 MB 00:03 2023-12-04T14:03:35.540 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:03:35.571 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:03:35.571 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:03:36.030 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:03:36.030 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:03:37.406 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:03:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-04T14:03:37.522 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-04T14:03:37.573 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-04T14:03:37.889 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-04T14:03:37.955 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librados2-2:18.0.0-7672.g7c730b29.el9.x86_64 5/31 2023-12-04T14:03:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:18.0.0-7672.g7c730b29.el9.x86_64 5/31 2023-12-04T14:03:38.123 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-2:18.0.0-7672.g7c730b29.el9.x86_64 6/31 2023-12-04T14:03:38.148 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-2:18.0.0-7672.g7c730b29.el9.x86_64 7/31 2023-12-04T14:03:38.194 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-2:18.0.0-7672.g7c730b29.el9.x86_64 7/31 2023-12-04T14:03:38.222 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-04T14:03:38.258 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-argparse-2:18.0.0-7672.g7c730b29.el9. 9/31 2023-12-04T14:03:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-2:18.0.0-7672.g7c730b29.el9.x86_64 10/31 2023-12-04T14:03:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libradosstriper1-2:18.0.0-7672.g7c730b29.el9.x86_6 11/31 2023-12-04T14:03:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7672.g7c730b29.el9.x86_6 11/31 2023-12-04T14:03:38.505 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-04T14:03:38.548 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-04T14:03:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-04T14:03:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-04T14:03:38.668 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-04T14:03:38.700 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-04T14:03:39.005 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-04T14:03:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librbd1-2:18.0.0-7672.g7c730b29.el9.x86_64 19/31 2023-12-04T14:03:39.120 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:18.0.0-7672.g7c730b29.el9.x86_64 19/31 2023-12-04T14:03:39.152 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-2:18.0.0-7672.g7c730b29.el9.x86_64 20/31 2023-12-04T14:03:39.187 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-04T14:03:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-04T14:03:39.650 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librgw2-2:18.0.0-7672.g7c730b29.el9.x86_64 23/31 2023-12-04T14:03:39.712 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librgw2-2:18.0.0-7672.g7c730b29.el9.x86_64 23/31 2023-12-04T14:03:39.780 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-2:18.0.0-7672.g7c730b29.el9.x86_64 24/31 2023-12-04T14:03:39.810 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-common-2:18.0.0-7672.g7c730b29.el9.x8 25/31 2023-12-04T14:03:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7672.g7c730b29.el9.x86_64 28/31 2023-12-04T14:03:40.039 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2023-12-04T14:03:40.039 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2023-12-04T14:03:40.039 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2023-12-04T14:03:40.039 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2023-12-04T14:03:40.039 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2023-12-04T14:03:40.039 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /root 2023-12-04T14:03:40.039 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2023-12-04T14:03:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:40.845 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:18.0.0-7672.g7c730b29.el9.x86_64 29/31 2023-12-04T14:03:40.867 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7672.g7c730b29.el9.x86_64 29/31 2023-12-04T14:03:40.867 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:03:40.867 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-04T14:03:40.867 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-04T14:03:40.867 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-04T14:03:40.867 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:40.869 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-04T14:03:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-04T14:03:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-04T14:03:41.888 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:18.0.0-7672.g7c730b29.el9.x86_64 26/31 2023-12-04T14:03:41.923 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-2:18.0.0-7672.g7c730b29.el9.x86_64 26/31 2023-12-04T14:03:42.038 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-04T14:03:42.038 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:18.0.0-7672.g7c730b29.el9.x86_64 1/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:18.0.0-7672.g7c730b29.el9.x86_64 2/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:18.0.0-7672.g7c730b29.el9.x86_64 3/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:18.0.0-7672.g7c730b29.el9.x86_64 4/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:18.0.0-7672.g7c730b29.el9.x86_64 5/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:18.0.0-7672.g7c730b29.el9.x86_6 6/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:18.0.0-7672.g7c730b29.el9.x86_64 7/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7672.g7c730b29.el9. 8/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:18.0.0-7672.g7c730b29.el9.x8 9/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:18.0.0-7672.g7c730b29.el9.x86_64 10/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:18.0.0-7672.g7c730b29.el9.x86_64 11/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:18.0.0-7672.g7c730b29.el9.x86_64 12/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:18.0.0-7672.g7c730b29.el9.x86_64 13/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-04T14:03:42.039 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-04T14:03:42.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-04T14:03:42.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-04T14:03:42.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-04T14:03:42.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-04T14:03:42.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-04T14:03:42.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-04T14:03:42.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-04T14:03:42.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-04T14:03:42.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-04T14:03:42.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-04T14:03:42.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-04T14:03:42.041 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-04T14:03:42.041 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:18.0.0-7672.g7c730b29.el9.x86_64 28/31 2023-12-04T14:03:42.041 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-04T14:03:42.041 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:18.0.0-7672.g7c730b29.el9.x86_64 30/31 2023-12-04T14:03:42.496 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:18.0.0-7672.g7c730b29.el9.x86_64 26/31 2023-12-04T14:03:42.543 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-2:18.0.0-7672.g7c730b29.el9.x86_64 27/31 2023-12-04T14:03:42.637 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-2:18.0.0-7672.g7c730b29.el9.x86_64 27/31 2023-12-04T14:03:42.637 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-04T14:03:42.637 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-04T14:03:42.637 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:03:42.677 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-selinux-2:18.0.0-7672.g7c730b29.el9.x86_64 28/31 2023-12-04T14:03:42.972 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-04T14:03:42.972 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:42.972 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2023-12-04T14:03:42.972 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.972 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.972 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:03:42.972 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-04T14:03:42.972 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.972 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-04T14:03:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-04T14:03:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-04T14:03:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: re2-1:20211101-3.el9.x86_64 2023-12-04T14:03:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-04T14:03:42.974 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:42.974 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:03:43.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-test 2023-12-04T14:03:43.786 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:27 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:03:43.936 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:03:43.936 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:03:43.936 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-12-04T14:03:43.936 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 41 M 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 167 k 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2023-12-04T14:03:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:43.938 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 42 M 2023-12-04T14:03:43.938 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 174 M 2023-12-04T14:03:43.938 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:03:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 186 kB 00:00 2023-12-04T14:03:44.144 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-04T14:03:44.186 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-04T14:03:44.228 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-04T14:03:44.345 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): libcephsqlite-18.0.0-7672.g7c730b29.el9. 411 kB/s | 167 kB 00:00 2023-12-04T14:03:46.045 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): ceph-test-18.0.0-7672.g7c730b29.el9.x86_ 20 MB/s | 41 MB 00:02 2023-12-04T14:03:46.048 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:03:46.048 INFO:teuthology.orchestra.run.smithi146.stdout:Total 20 MB/s | 42 MB 00:02 2023-12-04T14:03:46.048 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:03:46.069 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:03:46.069 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:03:46.410 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:03:46.410 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:03:46.948 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:03:47.021 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-04T14:03:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-04T14:03:47.094 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-04T14:03:47.137 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-04T14:03:47.164 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:18.0.0-7672.g7c730b29.el9.x86_64 5/6 2023-12-04T14:03:51.023 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7672.g7c730b29.el9.x86_64 5/6 2023-12-04T14:03:51.097 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:18.0.0-7672.g7c730b29.el9.x86_64 6/6 2023-12-04T14:03:52.369 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7672.g7c730b29.el9.x86_64 28/31 2023-12-04T14:03:52.369 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /sys 2023-12-04T14:03:52.369 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /proc 2023-12-04T14:03:52.369 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2023-12-04T14:03:52.370 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /var/tmp 2023-12-04T14:03:52.370 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /home 2023-12-04T14:03:52.370 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /root 2023-12-04T14:03:52.370 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /tmp 2023-12-04T14:03:52.370 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:03:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-test-2:18.0.0-7672.g7c730b29.el9.x86_64 6/6 2023-12-04T14:03:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:18.0.0-7672.g7c730b29.el9.x86_64 1/6 2023-12-04T14:03:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:18.0.0-7672.g7c730b29.el9.x86_64 2/6 2023-12-04T14:03:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-04T14:03:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-04T14:03:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-04T14:03:53.195 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-2:18.0.0-7672.g7c730b29.el9.x86_64 29/31 2023-12-04T14:03:53.225 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7672.g7c730b29.el9.x86_64 29/31 2023-12-04T14:03:53.225 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:03:53.225 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-04T14:03:53.225 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-04T14:03:53.226 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-04T14:03:53.226 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:03:53.227 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-04T14:03:53.306 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-04T14:03:53.306 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-04T14:03:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-04T14:03:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:53.494 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:03:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: jq-1.6-16.el9.x86_64 2023-12-04T14:03:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-04T14:03:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-04T14:03:53.495 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-04T14:03:53.495 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:53.495 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:03:53.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph 2023-12-04T14:03:54.364 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:38 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:03:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-04T14:03:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-2:18.0.0-7672.g7c730b29.el9.x86_64 1/31 2023-12-04T14:03:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-2:18.0.0-7672.g7c730b29.el9.x86_64 2/31 2023-12-04T14:03:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-2:18.0.0-7672.g7c730b29.el9.x86_64 3/31 2023-12-04T14:03:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-selinux-2:18.0.0-7672.g7c730b29.el9.x86_64 4/31 2023-12-04T14:03:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-2:18.0.0-7672.g7c730b29.el9.x86_64 5/31 2023-12-04T14:03:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libradosstriper1-2:18.0.0-7672.g7c730b29.el9.x86_6 6/31 2023-12-04T14:03:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librgw2-2:18.0.0-7672.g7c730b29.el9.x86_64 7/31 2023-12-04T14:03:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7672.g7c730b29.el9. 8/31 2023-12-04T14:03:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-common-2:18.0.0-7672.g7c730b29.el9.x8 9/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-2:18.0.0-7672.g7c730b29.el9.x86_64 10/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-2:18.0.0-7672.g7c730b29.el9.x86_64 11/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-2:18.0.0-7672.g7c730b29.el9.x86_64 12/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-2:18.0.0-7672.g7c730b29.el9.x86_64 13/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-04T14:03:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-04T14:03:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-04T14:03:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-04T14:03:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-04T14:03:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-04T14:03:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:18.0.0-7672.g7c730b29.el9.x86_64 28/31 2023-12-04T14:03:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-04T14:03:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:18.0.0-7672.g7c730b29.el9.x86_64 30/31 2023-12-04T14:03:54.511 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:03:54.513 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2023-12-04T14:03:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-12-04T14:03:54.514 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2023-12-04T14:03:54.514 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:03:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 6.6 k 2023-12-04T14:03:54.514 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-12-04T14:03:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 2.1 M 2023-12-04T14:03:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 1.5 M 2023-12-04T14:03:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 243 k 2023-12-04T14:03:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 4.4 M 2023-12-04T14:03:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 18 M 2023-12-04T14:03:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-04T14:03:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-04T14:03:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-04T14:03:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-04T14:03:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-04T14:03:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-04T14:03:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-04T14:03:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-04T14:03:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-04T14:03:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-04T14:03:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-04T14:03:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-04T14:03:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-04T14:03:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-04T14:03:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-04T14:03:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-04T14:03:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-04T14:03:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-04T14:03:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-04T14:03:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-04T14:03:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-04T14:03:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-04T14:03:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-04T14:03:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-04T14:03:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-04T14:03:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-04T14:03:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-04T14:03:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-04T14:03:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-04T14:03:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-04T14:03:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:54.518 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:03:54.518 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2023-12-04T14:03:54.519 INFO:teuthology.orchestra.run.smithi146.stdout:Install 36 Packages 2023-12-04T14:03:54.519 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:03:54.519 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 30 M 2023-12-04T14:03:54.520 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 112 M 2023-12-04T14:03:54.520 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:03:54.879 INFO:teuthology.orchestra.run.smithi146.stdout:(1/36): ceph-18.0.0-7672.g7c730b29.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2023-12-04T14:03:55.146 INFO:teuthology.orchestra.run.smithi146.stdout:(2/36): ceph-mds-18.0.0-7672.g7c730b29.el9.x86_ 4.8 MB/s | 2.1 MB 00:00 2023-12-04T14:03:55.330 INFO:teuthology.orchestra.run.smithi146.stdout:(3/36): ceph-mgr-18.0.0-7672.g7c730b29.el9.x86_ 2.4 MB/s | 1.5 MB 00:00 2023-12-04T14:03:55.413 INFO:teuthology.orchestra.run.smithi146.stdout:(4/36): ceph-mon-18.0.0-7672.g7c730b29.el9.x86_ 8.2 MB/s | 4.4 MB 00:00 2023-12-04T14:03:55.438 INFO:teuthology.orchestra.run.smithi146.stdout:(5/36): ceph-mgr-modules-core-18.0.0-7672.g7c73 2.2 MB/s | 243 kB 00:00 2023-12-04T14:03:55.564 INFO:teuthology.orchestra.run.smithi146.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-12-04T14:03:55.589 INFO:teuthology.orchestra.run.smithi146.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2023-12-04T14:03:55.590 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-04T14:03:55.590 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:03:55.590 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2023-12-04T14:03:55.590 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.590 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.590 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:03:55.590 INFO:teuthology.orchestra.run.smithi176.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-04T14:03:55.590 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-04T14:03:55.591 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-04T14:03:55.592 INFO:teuthology.orchestra.run.smithi176.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-04T14:03:55.592 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.592 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.592 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.592 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-04T14:03:55.592 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.592 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.592 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:03:55.592 INFO:teuthology.orchestra.run.smithi176.stdout: re2-1:20211101-3.el9.x86_64 2023-12-04T14:03:55.592 INFO:teuthology.orchestra.run.smithi176.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-04T14:03:55.592 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:03:55.592 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:03:55.614 INFO:teuthology.orchestra.run.smithi146.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 630 kB/s | 32 kB 00:00 2023-12-04T14:03:55.640 INFO:teuthology.orchestra.run.smithi146.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2023-12-04T14:03:55.665 INFO:teuthology.orchestra.run.smithi146.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2023-12-04T14:03:55.690 INFO:teuthology.orchestra.run.smithi146.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-04T14:03:55.740 INFO:teuthology.orchestra.run.smithi146.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2023-12-04T14:03:55.757 INFO:teuthology.orchestra.run.smithi146.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2023-12-04T14:03:55.783 INFO:teuthology.orchestra.run.smithi146.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2023-12-04T14:03:55.808 INFO:teuthology.orchestra.run.smithi146.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-04T14:03:55.833 INFO:teuthology.orchestra.run.smithi146.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2023-12-04T14:03:55.900 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-test 2023-12-04T14:03:55.908 INFO:teuthology.orchestra.run.smithi146.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2023-12-04T14:03:56.017 INFO:teuthology.orchestra.run.smithi146.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 405 kB/s | 43 kB 00:00 2023-12-04T14:03:56.042 INFO:teuthology.orchestra.run.smithi146.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 723 kB/s | 151 kB 00:00 2023-12-04T14:03:56.084 INFO:teuthology.orchestra.run.smithi146.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2023-12-04T14:03:56.250 INFO:teuthology.orchestra.run.smithi146.stdout:(21/36): ceph-osd-18.0.0-7672.g7c730b29.el9.x86 16 MB/s | 18 MB 00:01 2023-12-04T14:03:56.275 INFO:teuthology.orchestra.run.smithi146.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2023-12-04T14:03:56.292 INFO:teuthology.orchestra.run.smithi146.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 52 kB/s | 11 kB 00:00 2023-12-04T14:03:56.309 INFO:teuthology.orchestra.run.smithi146.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2023-12-04T14:03:56.342 INFO:teuthology.orchestra.run.smithi146.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2023-12-04T14:03:56.359 INFO:teuthology.orchestra.run.smithi146.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2023-12-04T14:03:56.376 INFO:teuthology.orchestra.run.smithi146.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2023-12-04T14:03:56.401 INFO:teuthology.orchestra.run.smithi146.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2023-12-04T14:03:56.426 INFO:teuthology.orchestra.run.smithi146.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2023-12-04T14:03:56.452 INFO:teuthology.orchestra.run.smithi146.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2023-12-04T14:03:56.468 INFO:teuthology.orchestra.run.smithi146.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2023-12-04T14:03:56.481 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:27 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:03:56.502 INFO:teuthology.orchestra.run.smithi146.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2023-12-04T14:03:56.528 INFO:teuthology.orchestra.run.smithi146.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2023-12-04T14:03:56.553 INFO:teuthology.orchestra.run.smithi146.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2023-12-04T14:03:56.578 INFO:teuthology.orchestra.run.smithi146.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2023-12-04T14:03:56.595 INFO:teuthology.orchestra.run.smithi146.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 485 kB/s | 20 kB 00:00 2023-12-04T14:03:56.600 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:03:56.600 INFO:teuthology.orchestra.run.smithi146.stdout:Total 15 MB/s | 30 MB 00:02 2023-12-04T14:03:56.610 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:03:56.626 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:03:56.626 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 41 M 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 167 k 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:03:56.627 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:03:56.628 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2023-12-04T14:03:56.628 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:03:56.628 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 42 M 2023-12-04T14:03:56.628 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 174 M 2023-12-04T14:03:56.628 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:03:56.649 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:03:56.649 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:03:56.784 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2023-12-04T14:03:56.826 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-04T14:03:56.868 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-04T14:03:56.893 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2023-12-04T14:03:57.027 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): libcephsqlite-18.0.0-7672.g7c730b29.el9. 419 kB/s | 167 kB 00:00 2023-12-04T14:03:57.054 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:03:57.055 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:03:58.443 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:03:58.490 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-04T14:03:58.522 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-04T14:03:58.550 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-04T14:03:58.620 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-04T14:03:58.691 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-04T14:03:58.723 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-04T14:03:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-04T14:03:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-04T14:03:58.917 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-04T14:03:58.990 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-04T14:03:59.043 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-04T14:03:59.081 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-04T14:03:59.130 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-04T14:03:59.189 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-04T14:03:59.293 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): ceph-test-18.0.0-7672.g7c730b29.el9.x86_ 15 MB/s | 41 MB 00:02 2023-12-04T14:03:59.297 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:03:59.297 INFO:teuthology.orchestra.run.smithi176.stdout:Total 16 MB/s | 42 MB 00:02 2023-12-04T14:03:59.297 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:03:59.316 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:03:59.316 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:03:59.388 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-04T14:03:59.435 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-04T14:03:59.466 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-04T14:03:59.500 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-04T14:03:59.530 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-04T14:03:59.606 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-04T14:03:59.649 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-04T14:03:59.664 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:03:59.664 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:03:59.682 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-04T14:03:59.722 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-04T14:03:59.766 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-04T14:03:59.809 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-04T14:03:59.899 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-04T14:03:59.931 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-04T14:04:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-04T14:04:00.142 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-04T14:04:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:04:00.264 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-04T14:04:00.286 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-04T14:04:00.326 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-04T14:04:00.359 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-04T14:04:00.411 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-04T14:04:00.418 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7672.g7c730b29.el9. 31/36 2023-12-04T14:04:00.438 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephsqlite-2:18.0.0-7672.g7c730b29.el9.x86_64 5/6 2023-12-04T14:04:00.444 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:18.0.0-7672.g7c730b29.el9.x86_64 32/36 2023-12-04T14:04:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7672.g7c730b29.el9.x86_64 32/36 2023-12-04T14:04:00.470 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:00.470 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-04T14:04:00.470 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-04T14:04:00.470 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-04T14:04:00.470 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:18.0.0-7672.g7c730b29.el9.x86_64 33/36 2023-12-04T14:04:02.091 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:18.0.0-7672.g7c730b29.el9.x86_64 33/36 2023-12-04T14:04:02.091 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:02.091 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-04T14:04:02.091 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-04T14:04:02.091 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-04T14:04:02.091 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:02.563 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:18.0.0-7672.g7c730b29.el9.x86_64 34/36 2023-12-04T14:04:02.587 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:18.0.0-7672.g7c730b29.el9.x86_64 34/36 2023-12-04T14:04:02.587 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:02.587 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-04T14:04:02.587 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-04T14:04:02.587 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-04T14:04:02.587 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:02.789 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:18.0.0-7672.g7c730b29.el9.x86_64 35/36 2023-12-04T14:04:02.812 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:18.0.0-7672.g7c730b29.el9.x86_64 35/36 2023-12-04T14:04:02.812 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:02.812 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-04T14:04:02.812 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-04T14:04:02.812 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-04T14:04:02.812 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:02.869 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:18.0.0-7672.g7c730b29.el9.x86_64 36/36 2023-12-04T14:04:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7672.g7c730b29.el9.x86_64 5/6 2023-12-04T14:04:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-2:18.0.0-7672.g7c730b29.el9.x86_64 6/6 2023-12-04T14:04:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-2:18.0.0-7672.g7c730b29.el9.x86_64 36/36 2023-12-04T14:04:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:18.0.0-7672.g7c730b29.el9.x86_64 1/36 2023-12-04T14:04:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:18.0.0-7672.g7c730b29.el9.x86_64 2/36 2023-12-04T14:04:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:18.0.0-7672.g7c730b29.el9.x86_64 3/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:18.0.0-7672.g7c730b29.el9.x86_64 4/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:18.0.0-7672.g7c730b29.el9.x86_64 5/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7672.g7c730b29.el9. 6/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-04T14:04:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-04T14:04:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-04T14:04:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-04T14:04:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-04T14:04:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-04T14:04:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-04T14:04:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-04T14:04:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: lua-5.4.4-4.el9.x86_64 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-04T14:04:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-04T14:04:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply-3.11-14.el9.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-04T14:04:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-04T14:04:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:05.599 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:05.890 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-base 2023-12-04T14:04:06.462 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:50 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:06.464 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-test-2:18.0.0-7672.g7c730b29.el9.x86_64 6/6 2023-12-04T14:04:06.465 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-2:18.0.0-7672.g7c730b29.el9.x86_64 1/6 2023-12-04T14:04:06.465 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephsqlite-2:18.0.0-7672.g7c730b29.el9.x86_64 2/6 2023-12-04T14:04:06.465 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-04T14:04:06.465 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-04T14:04:06.465 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-04T14:04:06.557 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-base-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:04:06.606 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:06.607 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-12-04T14:04:06.607 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:06.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install cephadm 2023-12-04T14:04:06.847 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-04T14:04:06.847 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:06.847 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:04:06.848 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:06.848 INFO:teuthology.orchestra.run.smithi176.stdout: jq-1.6-16.el9.x86_64 2023-12-04T14:04:06.848 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:06.848 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-04T14:04:06.848 INFO:teuthology.orchestra.run.smithi176.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-04T14:04:06.848 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-04T14:04:06.848 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:06.848 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:07.135 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph 2023-12-04T14:04:07.251 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:51 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:07.396 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:07.397 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:07.397 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-12-04T14:04:07.397 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:07.397 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:07.397 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 775 k 2023-12-04T14:04:07.397 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:07.397 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:07.397 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:07.397 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-12-04T14:04:07.397 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:07.398 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 775 k 2023-12-04T14:04:07.398 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 790 k 2023-12-04T14:04:07.398 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:07.688 INFO:teuthology.orchestra.run.smithi146.stdout:cephadm-18.0.0-7672.g7c730b29.el9.noarch.rpm 2.6 MB/s | 775 kB 00:00 2023-12-04T14:04:07.688 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:07.688 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.6 MB/s | 775 kB 00:00 2023-12-04T14:04:07.688 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:07.693 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:07.693 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:07.699 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:07.699 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:07.718 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:38 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:07.776 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:07.870 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:07.873 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2023-12-04T14:04:07.873 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-04T14:04:07.873 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2023-12-04T14:04:07.873 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:04:07.873 INFO:teuthology.orchestra.run.smithi176.stdout: ceph x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 6.6 k 2023-12-04T14:04:07.873 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-04T14:04:07.873 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 2.1 M 2023-12-04T14:04:07.873 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 1.5 M 2023-12-04T14:04:07.873 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 243 k 2023-12-04T14:04:07.873 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 4.4 M 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 18 M 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-04T14:04:07.874 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-04T14:04:07.875 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-04T14:04:07.876 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-04T14:04:07.876 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-04T14:04:07.876 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:07.876 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:04:07.876 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2023-12-04T14:04:07.876 INFO:teuthology.orchestra.run.smithi176.stdout:Install 36 Packages 2023-12-04T14:04:07.876 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:07.876 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 30 M 2023-12-04T14:04:07.876 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 112 M 2023-12-04T14:04:07.877 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:04:08.146 INFO:teuthology.orchestra.run.smithi176.stdout:(1/36): ceph-18.0.0-7672.g7c730b29.el9.x86_64.r 43 kB/s | 6.6 kB 00:00 2023-12-04T14:04:08.211 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:08.388 INFO:teuthology.orchestra.run.smithi176.stdout:(2/36): ceph-mds-18.0.0-7672.g7c730b29.el9.x86_ 5.3 MB/s | 2.1 MB 00:00 2023-12-04T14:04:08.671 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:08.796 INFO:teuthology.orchestra.run.smithi176.stdout:(3/36): ceph-mon-18.0.0-7672.g7c730b29.el9.x86_ 6.8 MB/s | 4.4 MB 00:00 2023-12-04T14:04:08.888 INFO:teuthology.orchestra.run.smithi176.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7672.g7c73 2.6 MB/s | 243 kB 00:00 2023-12-04T14:04:08.978 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:08.978 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:08.979 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:08.979 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:08.979 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:08.979 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:09.005 INFO:teuthology.orchestra.run.smithi176.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-12-04T14:04:09.047 INFO:teuthology.orchestra.run.smithi176.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-12-04T14:04:09.080 INFO:teuthology.orchestra.run.smithi176.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2023-12-04T14:04:09.114 INFO:teuthology.orchestra.run.smithi176.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-04T14:04:09.148 INFO:teuthology.orchestra.run.smithi176.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-04T14:04:09.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-immutable-object-cache 2023-12-04T14:04:09.323 INFO:teuthology.orchestra.run.smithi176.stdout:(10/36): ceph-osd-18.0.0-7672.g7c730b29.el9.x86 19 MB/s | 18 MB 00:00 2023-12-04T14:04:09.348 INFO:teuthology.orchestra.run.smithi176.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 241 kB 00:00 2023-12-04T14:04:09.382 INFO:teuthology.orchestra.run.smithi176.stdout:(12/36): ceph-mgr-18.0.0-7672.g7c730b29.el9.x86 1.1 MB/s | 1.5 MB 00:01 2023-12-04T14:04:09.416 INFO:teuthology.orchestra.run.smithi176.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 13 MB/s | 1.1 MB 00:00 2023-12-04T14:04:09.441 INFO:teuthology.orchestra.run.smithi176.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2023-12-04T14:04:09.466 INFO:teuthology.orchestra.run.smithi176.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2023-12-04T14:04:09.491 INFO:teuthology.orchestra.run.smithi176.stdout:(16/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-04T14:04:09.517 INFO:teuthology.orchestra.run.smithi176.stdout:(17/36): python3-urllib3-1.26.5-4.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2023-12-04T14:04:09.768 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:53 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:09.817 INFO:teuthology.orchestra.run.smithi176.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2023-12-04T14:04:09.884 INFO:teuthology.orchestra.run.smithi176.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 412 kB/s | 151 kB 00:00 2023-12-04T14:04:09.915 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:09.915 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:09.915 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2023-12-04T14:04:09.915 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:09.915 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 142 k 2023-12-04T14:04:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:09.916 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:09.916 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:09.916 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-12-04T14:04:09.916 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:09.916 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 142 k 2023-12-04T14:04:09.916 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 399 k 2023-12-04T14:04:09.917 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:10.001 INFO:teuthology.orchestra.run.smithi176.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 936 kB/s | 172 kB 00:00 2023-12-04T14:04:10.043 INFO:teuthology.orchestra.run.smithi176.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2023-12-04T14:04:10.076 INFO:teuthology.orchestra.run.smithi176.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2023-12-04T14:04:10.118 INFO:teuthology.orchestra.run.smithi176.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2023-12-04T14:04:10.142 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-immutable-object-cache-18.0.0-7672.g7c730b 629 kB/s | 142 kB 00:00 2023-12-04T14:04:10.143 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:10.143 INFO:teuthology.orchestra.run.smithi146.stdout:Total 625 kB/s | 142 kB 00:00 2023-12-04T14:04:10.143 INFO:teuthology.orchestra.run.smithi176.stdout:(24/36): lua-devel-5.4.4-4.el9.x86_64.rpm 33 kB/s | 22 kB 00:00 2023-12-04T14:04:10.144 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:10.151 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:10.151 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:10.168 INFO:teuthology.orchestra.run.smithi176.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2023-12-04T14:04:10.194 INFO:teuthology.orchestra.run.smithi176.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 260 kB/s | 19 kB 00:00 2023-12-04T14:04:10.203 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:10.203 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:10.253 INFO:teuthology.orchestra.run.smithi176.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 554 kB/s | 46 kB 00:00 2023-12-04T14:04:10.286 INFO:teuthology.orchestra.run.smithi176.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2023-12-04T14:04:10.340 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:10.370 INFO:teuthology.orchestra.run.smithi176.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2023-12-04T14:04:10.395 INFO:teuthology.orchestra.run.smithi176.stdout:(30/36): python3-jaraco-text-3.2.0-6.el9.noarch 78 kB/s | 20 kB 00:00 2023-12-04T14:04:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7672.g7c730b29. 1/1 2023-12-04T14:04:10.420 INFO:teuthology.orchestra.run.smithi176.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2023-12-04T14:04:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7672.g7c730b29. 1/1 2023-12-04T14:04:10.426 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:10.426 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-04T14:04:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:10.454 INFO:teuthology.orchestra.run.smithi176.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2023-12-04T14:04:10.521 INFO:teuthology.orchestra.run.smithi176.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 285 kB/s | 36 kB 00:00 2023-12-04T14:04:10.546 INFO:teuthology.orchestra.run.smithi176.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2023-12-04T14:04:10.571 INFO:teuthology.orchestra.run.smithi176.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2023-12-04T14:04:10.605 INFO:teuthology.orchestra.run.smithi176.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2023-12-04T14:04:10.611 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:10.611 INFO:teuthology.orchestra.run.smithi176.stdout:Total 11 MB/s | 30 MB 00:02 2023-12-04T14:04:10.624 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:04:10.665 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:04:10.665 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:04:11.085 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:04:11.085 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:04:11.415 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7672.g7c730b29. 1/1 2023-12-04T14:04:11.415 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:11.415 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:11.415 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:11.415 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:11.415 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:11.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr 2023-12-04T14:04:12.236 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:56 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:12.332 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-mgr-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:04:12.382 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:12.383 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-12-04T14:04:12.383 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:12.423 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:04:12.446 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-dashboard 2023-12-04T14:04:12.483 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-04T14:04:12.515 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-04T14:04:12.552 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-04T14:04:12.623 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-04T14:04:12.667 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-04T14:04:12.700 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-04T14:04:12.792 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-04T14:04:12.862 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-04T14:04:12.902 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-04T14:04:12.974 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-04T14:04:13.024 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:57 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:13.027 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-04T14:04:13.075 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-04T14:04:13.123 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-04T14:04:13.170 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:13.171 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:13.171 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 1.8 M 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 28 k 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 15 k 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:13.172 INFO:teuthology.orchestra.run.smithi146.stdout:Install 7 Packages 2023-12-04T14:04:13.173 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:13.173 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 2.1 M 2023-12-04T14:04:13.173 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 8.7 M 2023-12-04T14:04:13.173 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:13.186 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-04T14:04:13.380 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-04T14:04:13.429 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-04T14:04:13.459 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-04T14:04:13.502 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-04T14:04:13.532 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-04T14:04:13.608 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-04T14:04:13.642 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-04T14:04:13.677 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-04T14:04:13.712 INFO:teuthology.orchestra.run.smithi146.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7672.g7c7 151 kB/s | 28 kB 00:00 2023-12-04T14:04:13.716 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-04T14:04:13.751 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-04T14:04:13.785 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-04T14:04:13.820 INFO:teuthology.orchestra.run.smithi146.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7672.g7c73 51 kB/s | 15 kB 00:00 2023-12-04T14:04:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-04T14:04:13.887 INFO:teuthology.orchestra.run.smithi146.stdout:(3/7): ceph-mgr-dashboard-18.0.0-7672.g7c730b29 4.9 MB/s | 1.8 MB 00:00 2023-12-04T14:04:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-04T14:04:13.937 INFO:teuthology.orchestra.run.smithi146.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 40 kB/s | 9.0 kB 00:00 2023-12-04T14:04:13.980 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-04T14:04:14.038 INFO:teuthology.orchestra.run.smithi146.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 243 kB/s | 31 kB 00:00 2023-12-04T14:04:14.113 INFO:teuthology.orchestra.run.smithi146.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 140 kB/s | 41 kB 00:00 2023-12-04T14:04:14.126 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-04T14:04:14.250 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-04T14:04:14.329 INFO:teuthology.orchestra.run.smithi146.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 481 kB/s | 188 kB 00:00 2023-12-04T14:04:14.334 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:14.334 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.8 MB/s | 2.1 MB 00:01 2023-12-04T14:04:14.334 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:14.342 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:14.342 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:14.402 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7672.g7c730b29.el9. 31/36 2023-12-04T14:04:14.402 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:14.403 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:14.429 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-2:18.0.0-7672.g7c730b29.el9.x86_64 32/36 2023-12-04T14:04:14.453 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7672.g7c730b29.el9.x86_64 32/36 2023-12-04T14:04:14.453 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:14.453 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-04T14:04:14.453 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-04T14:04:14.453 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-04T14:04:14.453 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:14.704 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:14.764 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-04T14:04:14.798 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-04T14:04:14.845 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-04T14:04:14.876 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-04T14:04:14.906 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7672.g7c730b29.el9.n 5/7 2023-12-04T14:04:15.147 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7672.g7c730b29.el9. 6/7 2023-12-04T14:04:15.174 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7672.g7c730b29.el9.noarc 7/7 2023-12-04T14:04:15.722 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7672.g7c730b29.el9.noarc 7/7 2023-12-04T14:04:15.722 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7672.g7c730b29.el9. 1/7 2023-12-04T14:04:15.722 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7672.g7c730b29.el9.noarc 2/7 2023-12-04T14:04:15.722 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7672.g7c730b29.el9.n 3/7 2023-12-04T14:04:15.722 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-04T14:04:15.722 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-04T14:04:15.722 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-04T14:04:16.038 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-04T14:04:16.038 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:16.039 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:16.039 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:16.039 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:16.039 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:16.039 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-04T14:04:16.039 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-04T14:04:16.039 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-04T14:04:16.039 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-04T14:04:16.039 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:16.039 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:16.040 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-2:18.0.0-7672.g7c730b29.el9.x86_64 33/36 2023-12-04T14:04:16.067 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-2:18.0.0-7672.g7c730b29.el9.x86_64 33/36 2023-12-04T14:04:16.067 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:16.067 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-04T14:04:16.067 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-04T14:04:16.067 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-04T14:04:16.067 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:16.267 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-04T14:04:16.523 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-2:18.0.0-7672.g7c730b29.el9.x86_64 34/36 2023-12-04T14:04:16.546 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-2:18.0.0-7672.g7c730b29.el9.x86_64 34/36 2023-12-04T14:04:16.546 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:16.546 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-04T14:04:16.547 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-04T14:04:16.547 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-04T14:04:16.547 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:16.749 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-2:18.0.0-7672.g7c730b29.el9.x86_64 35/36 2023-12-04T14:04:16.772 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-2:18.0.0-7672.g7c730b29.el9.x86_64 35/36 2023-12-04T14:04:16.772 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:16.772 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-04T14:04:16.772 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-04T14:04:16.772 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-04T14:04:16.772 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:16.829 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-2:18.0.0-7672.g7c730b29.el9.x86_64 36/36 2023-12-04T14:04:16.845 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:00 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:16.995 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:16.996 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2023-12-04T14:04:16.996 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-12-04T14:04:16.996 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2023-12-04T14:04:16.996 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:16.996 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 7.4 M 2023-12-04T14:04:16.996 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-12-04T14:04:16.996 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-04T14:04:16.996 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-04T14:04:16.996 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-04T14:04:16.996 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-04T14:04:16.997 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-04T14:04:16.997 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-04T14:04:16.997 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-04T14:04:16.997 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-04T14:04:16.997 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-04T14:04:16.997 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-04T14:04:16.997 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-04T14:04:16.997 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:16.997 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:16.997 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2023-12-04T14:04:16.997 INFO:teuthology.orchestra.run.smithi146.stdout:Install 12 Packages 2023-12-04T14:04:16.997 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:16.998 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 37 M 2023-12-04T14:04:16.998 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 209 M 2023-12-04T14:04:16.998 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:17.411 INFO:teuthology.orchestra.run.smithi146.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 237 kB/s | 32 kB 00:00 2023-12-04T14:04:17.445 INFO:teuthology.orchestra.run.smithi146.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 539 kB/s | 17 kB 00:00 2023-12-04T14:04:17.537 INFO:teuthology.orchestra.run.smithi146.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-12-04T14:04:17.562 INFO:teuthology.orchestra.run.smithi146.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 295 kB/s | 34 kB 00:00 2023-12-04T14:04:17.762 INFO:teuthology.orchestra.run.smithi146.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2023-12-04T14:04:17.846 INFO:teuthology.orchestra.run.smithi146.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2023-12-04T14:04:17.863 INFO:teuthology.orchestra.run.smithi146.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2023-12-04T14:04:18.038 INFO:teuthology.orchestra.run.smithi146.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2023-12-04T14:04:18.263 INFO:teuthology.orchestra.run.smithi146.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2023-12-04T14:04:18.280 INFO:teuthology.orchestra.run.smithi146.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 814 kB/s | 195 kB 00:00 2023-12-04T14:04:18.439 INFO:teuthology.orchestra.run.smithi146.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2023-12-04T14:04:18.622 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-2:18.0.0-7672.g7c730b29.el9.x86_64 36/36 2023-12-04T14:04:18.622 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-2:18.0.0-7672.g7c730b29.el9.x86_64 1/36 2023-12-04T14:04:18.622 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-2:18.0.0-7672.g7c730b29.el9.x86_64 2/36 2023-12-04T14:04:18.622 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-2:18.0.0-7672.g7c730b29.el9.x86_64 3/36 2023-12-04T14:04:18.622 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-2:18.0.0-7672.g7c730b29.el9.x86_64 4/36 2023-12-04T14:04:18.622 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-2:18.0.0-7672.g7c730b29.el9.x86_64 5/36 2023-12-04T14:04:18.622 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7672.g7c730b29.el9. 6/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-04T14:04:18.623 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-04T14:04:18.624 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-04T14:04:18.625 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-04T14:04:18.855 INFO:teuthology.orchestra.run.smithi146.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 4.7 MB/s | 7.4 MB 00:01 2023-12-04T14:04:18.858 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:18.858 INFO:teuthology.orchestra.run.smithi146.stdout:Total 20 MB/s | 37 MB 00:01 2023-12-04T14:04:18.871 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:18.894 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:18.894 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:19.284 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:19.284 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:19.632 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-04T14:04:19.632 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:19.632 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:04:19.632 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:19.632 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:19.632 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:19.632 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: lua-5.4.4-4.el9.x86_64 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-04T14:04:19.633 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply-3.11-14.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-04T14:04:19.634 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-04T14:04:19.635 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-04T14:04:19.635 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-04T14:04:19.635 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-04T14:04:19.635 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:19.635 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:19.980 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-base 2023-12-04T14:04:19.986 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:20.054 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-04T14:04:20.093 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-04T14:04:20.189 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-04T14:04:20.221 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-04T14:04:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-04T14:04:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-04T14:04:20.506 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-04T14:04:20.555 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:51 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:20.650 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-base-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:04:20.700 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:20.701 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-04T14:04:20.701 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:20.758 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install cephadm 2023-12-04T14:04:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-04T14:04:21.275 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-04T14:04:21.336 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:52 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:21.480 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:21.481 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:21.481 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-04T14:04:21.481 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:21.481 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:04:21.481 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 775 k 2023-12-04T14:04:21.482 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:21.482 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:04:21.482 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:21.482 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-04T14:04:21.482 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:21.482 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 775 k 2023-12-04T14:04:21.482 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 790 k 2023-12-04T14:04:21.482 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:04:21.973 INFO:teuthology.orchestra.run.smithi176.stdout:cephadm-18.0.0-7672.g7c730b29.el9.noarch.rpm 1.5 MB/s | 775 kB 00:00 2023-12-04T14:04:21.973 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:21.974 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.5 MB/s | 775 kB 00:00 2023-12-04T14:04:21.974 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:04:21.979 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:04:21.979 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:04:21.985 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:04:21.985 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:04:22.059 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:04:22.531 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:22.550 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-04T14:04:22.619 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:22.990 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:23.332 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:23.333 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:23.333 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:04:23.333 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:23.333 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:23.333 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:23.559 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-immutable-object-cache 2023-12-04T14:04:23.586 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-04T14:04:23.610 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7672.g7c730 12/12 2023-12-04T14:04:24.145 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:55 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:24.295 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:24.296 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:24.296 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2023-12-04T14:04:24.296 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:24.296 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:04:24.296 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 142 k 2023-12-04T14:04:24.296 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:24.296 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:04:24.296 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:24.296 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-04T14:04:24.296 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:24.297 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 142 k 2023-12-04T14:04:24.297 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 399 k 2023-12-04T14:04:24.297 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:04:24.645 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-immutable-object-cache-18.0.0-7672.g7c730b 407 kB/s | 142 kB 00:00 2023-12-04T14:04:24.646 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:24.646 INFO:teuthology.orchestra.run.smithi176.stdout:Total 405 kB/s | 142 kB 00:00 2023-12-04T14:04:24.646 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:04:24.654 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:04:24.654 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:04:24.705 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:04:24.705 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:04:24.847 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:04:24.904 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7672.g7c730b29. 1/1 2023-12-04T14:04:24.928 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7672.g7c730b29. 1/1 2023-12-04T14:04:24.928 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:24.928 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-04T14:04:24.928 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7672.g7c730b29. 1/1 2023-12-04T14:04:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:25.902 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:04:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:25.902 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:26.060 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7672.g7c730 12/12 2023-12-04T14:04:26.060 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7672.g7c730 1/12 2023-12-04T14:04:26.060 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-04T14:04:26.060 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-04T14:04:26.060 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-04T14:04:26.060 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-04T14:04:26.060 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-04T14:04:26.060 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-04T14:04:26.060 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-04T14:04:26.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-04T14:04:26.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-04T14:04:26.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-04T14:04:26.131 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr 2023-12-04T14:04:26.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-04T14:04:26.484 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-04T14:04:26.485 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-04T14:04:26.485 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:26.485 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:26.716 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:57 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:26.811 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-mgr-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:04:26.846 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-rook 2023-12-04T14:04:26.861 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:26.862 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-04T14:04:26.862 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:26.924 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-dashboard 2023-12-04T14:04:27.436 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:11 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:27.500 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:58 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:27.582 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:27.583 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2023-12-04T14:04:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-12-04T14:04:27.583 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2023-12-04T14:04:27.583 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 50 k 2023-12-04T14:04:27.583 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-12-04T14:04:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-04T14:04:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:27.584 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2023-12-04T14:04:27.585 INFO:teuthology.orchestra.run.smithi146.stdout:Install 13 Packages 2023-12-04T14:04:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:27.585 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 2.0 M 2023-12-04T14:04:27.585 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 26 M 2023-12-04T14:04:27.585 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:27.644 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:27.645 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:27.645 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 1.8 M 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 28 k 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 15 k 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout:Installing weak dependencies: 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:04:27.646 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:27.647 INFO:teuthology.orchestra.run.smithi176.stdout:Install 7 Packages 2023-12-04T14:04:27.647 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:27.647 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 2.1 M 2023-12-04T14:04:27.647 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 8.7 M 2023-12-04T14:04:27.647 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:04:27.760 INFO:teuthology.orchestra.run.smithi146.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 194 kB/s | 24 kB 00:00 2023-12-04T14:04:27.785 INFO:teuthology.orchestra.run.smithi146.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2023-12-04T14:04:27.835 INFO:teuthology.orchestra.run.smithi146.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-12-04T14:04:27.860 INFO:teuthology.orchestra.run.smithi146.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-12-04T14:04:27.886 INFO:teuthology.orchestra.run.smithi146.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-04T14:04:27.911 INFO:teuthology.orchestra.run.smithi146.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2023-12-04T14:04:27.936 INFO:teuthology.orchestra.run.smithi146.stdout:(7/13): ceph-mgr-rook-18.0.0-7672.g7c730b29.el9 166 kB/s | 50 kB 00:00 2023-12-04T14:04:27.949 INFO:teuthology.orchestra.run.smithi176.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7672.g7c7 114 kB/s | 28 kB 00:00 2023-12-04T14:04:27.982 INFO:teuthology.orchestra.run.smithi176.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7672.g7c73 54 kB/s | 15 kB 00:00 2023-12-04T14:04:28.095 INFO:teuthology.orchestra.run.smithi146.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 154 kB/s | 32 kB 00:00 2023-12-04T14:04:28.137 INFO:teuthology.orchestra.run.smithi146.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 66 kB/s | 15 kB 00:00 2023-12-04T14:04:28.212 INFO:teuthology.orchestra.run.smithi146.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 867 kB/s | 197 kB 00:00 2023-12-04T14:04:28.224 INFO:teuthology.orchestra.run.smithi176.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 33 kB/s | 9.0 kB 00:00 2023-12-04T14:04:28.250 INFO:teuthology.orchestra.run.smithi176.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 153 kB/s | 41 kB 00:00 2023-12-04T14:04:28.279 INFO:teuthology.orchestra.run.smithi146.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 416 kB/s | 59 kB 00:00 2023-12-04T14:04:28.312 INFO:teuthology.orchestra.run.smithi146.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2023-12-04T14:04:28.337 INFO:teuthology.orchestra.run.smithi146.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 718 kB/s | 90 kB 00:00 2023-12-04T14:04:28.343 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:28.343 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2023-12-04T14:04:28.343 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:28.350 INFO:teuthology.orchestra.run.smithi176.stdout:(5/7): ceph-mgr-dashboard-18.0.0-7672.g7c730b29 2.7 MB/s | 1.8 MB 00:00 2023-12-04T14:04:28.356 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:28.356 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:28.375 INFO:teuthology.orchestra.run.smithi176.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 204 kB/s | 31 kB 00:00 2023-12-04T14:04:28.456 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:28.456 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:28.508 INFO:teuthology.orchestra.run.smithi176.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 729 kB/s | 188 kB 00:00 2023-12-04T14:04:28.512 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:28.512 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.4 MB/s | 2.1 MB 00:00 2023-12-04T14:04:28.512 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:04:28.520 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:04:28.520 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:04:28.581 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:04:28.581 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:04:28.885 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:04:28.951 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-04T14:04:28.985 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-04T14:04:28.990 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:29.031 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-04T14:04:29.064 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-04T14:04:29.094 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7672.g7c730b29.el9.n 5/7 2023-12-04T14:04:29.161 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-04T14:04:29.205 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-04T14:04:29.261 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-04T14:04:29.298 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-04T14:04:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7672.g7c730b29.el9. 6/7 2023-12-04T14:04:29.339 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-04T14:04:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7672.g7c730b29.el9.noarc 7/7 2023-12-04T14:04:29.432 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-04T14:04:29.550 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-04T14:04:29.599 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-04T14:04:29.901 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7672.g7c730b29.el9.noarc 7/7 2023-12-04T14:04:29.902 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7672.g7c730b29.el9. 1/7 2023-12-04T14:04:29.902 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7672.g7c730b29.el9.noarc 2/7 2023-12-04T14:04:29.902 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7672.g7c730b29.el9.n 3/7 2023-12-04T14:04:29.902 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-04T14:04:29.902 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-04T14:04:29.902 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-04T14:04:30.224 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-04T14:04:30.224 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:30.225 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:04:30.225 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:30.225 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:30.225 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:30.225 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-04T14:04:30.225 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-04T14:04:30.225 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-04T14:04:30.225 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-04T14:04:30.225 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:30.225 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:30.326 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-04T14:04:30.371 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-04T14:04:30.404 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-04T14:04:30.451 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-04T14:04:30.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-04T14:04:30.474 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:18.0.0-7672.g7c730b29.el9.noarch 13/13 2023-12-04T14:04:31.042 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:02 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:31.190 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:31.191 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2023-12-04T14:04:31.191 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-04T14:04:31.191 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2023-12-04T14:04:31.192 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:04:31.192 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 7.4 M 2023-12-04T14:04:31.192 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-04T14:04:31.192 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-04T14:04:31.192 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-04T14:04:31.192 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-04T14:04:31.192 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-04T14:04:31.192 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-04T14:04:31.192 INFO:teuthology.orchestra.run.smithi176.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-04T14:04:31.192 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-04T14:04:31.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-04T14:04:31.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-04T14:04:31.193 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-04T14:04:31.193 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-04T14:04:31.193 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:31.193 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:04:31.193 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2023-12-04T14:04:31.193 INFO:teuthology.orchestra.run.smithi176.stdout:Install 12 Packages 2023-12-04T14:04:31.193 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:31.193 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 37 M 2023-12-04T14:04:31.193 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 209 M 2023-12-04T14:04:31.194 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:04:31.327 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7672.g7c730b29.el9.noarch 13/13 2023-12-04T14:04:31.327 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7672.g7c730b29.el9.noarch 1/13 2023-12-04T14:04:31.327 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-04T14:04:31.327 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-04T14:04:31.327 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-04T14:04:31.327 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-04T14:04:31.327 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-04T14:04:31.327 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-04T14:04:31.327 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-04T14:04:31.327 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-04T14:04:31.327 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-04T14:04:31.328 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-04T14:04:31.328 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-04T14:04:31.374 INFO:teuthology.orchestra.run.smithi176.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 32 kB 00:00 2023-12-04T14:04:31.407 INFO:teuthology.orchestra.run.smithi176.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 537 kB/s | 17 kB 00:00 2023-12-04T14:04:31.449 INFO:teuthology.orchestra.run.smithi176.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 840 kB/s | 34 kB 00:00 2023-12-04T14:04:31.500 INFO:teuthology.orchestra.run.smithi176.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-12-04T14:04:31.683 INFO:teuthology.orchestra.run.smithi176.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2023-12-04T14:04:31.696 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-04T14:04:31.696 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:31.696 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:31.696 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:31.696 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:31.697 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:31.759 INFO:teuthology.orchestra.run.smithi176.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2023-12-04T14:04:31.784 INFO:teuthology.orchestra.run.smithi176.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-12-04T14:04:31.926 INFO:teuthology.orchestra.run.smithi176.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 5.6 MB/s | 803 kB 00:00 2023-12-04T14:04:31.932 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-cephadm 2023-12-04T14:04:32.084 INFO:teuthology.orchestra.run.smithi176.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2023-12-04T14:04:32.109 INFO:teuthology.orchestra.run.smithi176.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2023-12-04T14:04:32.235 INFO:teuthology.orchestra.run.smithi176.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2023-12-04T14:04:32.402 INFO:teuthology.orchestra.run.smithi176.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 6.3 MB/s | 7.4 MB 00:01 2023-12-04T14:04:32.404 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:32.405 INFO:teuthology.orchestra.run.smithi176.stdout:Total 31 MB/s | 37 MB 00:01 2023-12-04T14:04:32.418 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:04:32.442 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:04:32.442 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:04:32.519 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:16 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:32.665 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:32.666 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2023-12-04T14:04:32.666 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-12-04T14:04:32.666 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 133 k 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2023-12-04T14:04:32.667 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:32.668 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 6.8 M 2023-12-04T14:04:32.668 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 33 M 2023-12-04T14:04:32.668 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:32.839 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:04:32.839 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:04:32.989 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-12-04T14:04:33.080 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): ceph-mgr-cephadm-18.0.0-7672.g7c730b29.e 524 kB/s | 133 kB 00:00 2023-12-04T14:04:33.164 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-12-04T14:04:33.540 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:04:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-04T14:04:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-04T14:04:33.656 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 100 kB/s | 58 kB 00:00 2023-12-04T14:04:33.740 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 96 kB/s | 51 kB 00:00 2023-12-04T14:04:33.740 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-04T14:04:33.776 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-04T14:04:33.926 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 587 kB/s | 548 kB 00:00 2023-12-04T14:04:33.927 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:33.927 INFO:teuthology.orchestra.run.smithi146.stdout:Total 5.4 MB/s | 6.8 MB 00:01 2023-12-04T14:04:33.928 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:33.937 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:33.937 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:33.952 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-04T14:04:33.983 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-04T14:04:34.026 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:34.026 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:34.060 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-04T14:04:34.321 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-04T14:04:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-04T14:04:34.707 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-04T14:04:34.812 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-04T14:04:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-04T14:04:35.136 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-04T14:04:35.195 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-04T14:04:35.220 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 6/6 2023-12-04T14:04:36.056 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-04T14:04:36.097 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 6/6 2023-12-04T14:04:36.097 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 1/6 2023-12-04T14:04:36.098 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-04T14:04:36.098 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-04T14:04:36.098 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-04T14:04:36.098 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-04T14:04:36.414 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-04T14:04:36.415 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:36.415 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:36.415 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:36.415 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-04T14:04:36.415 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-04T14:04:36.415 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-04T14:04:36.415 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-04T14:04:36.415 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-04T14:04:36.415 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:36.415 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:36.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-fuse 2023-12-04T14:04:37.076 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-04T14:04:37.099 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7672.g7c730 12/12 2023-12-04T14:04:37.228 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:21 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:37.375 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:37.376 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-12-04T14:04:37.376 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:37.376 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:37.376 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 860 k 2023-12-04T14:04:37.376 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-12-04T14:04:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-04T14:04:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:37.377 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:37.377 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:37.377 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2023-12-04T14:04:37.377 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:37.377 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 939 k 2023-12-04T14:04:37.377 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 2.7 M 2023-12-04T14:04:37.377 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:37.499 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 646 kB/s | 79 kB 00:00 2023-12-04T14:04:37.699 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): ceph-fuse-18.0.0-7672.g7c730b29.el9.x86_ 2.6 MB/s | 860 kB 00:00 2023-12-04T14:04:37.700 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.8 MB/s | 939 kB 00:00 2023-12-04T14:04:37.701 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:37.722 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:37.796 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:37.797 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:37.989 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:38.132 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-04T14:04:38.208 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 2/2 2023-12-04T14:04:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 2/2 2023-12-04T14:04:38.934 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 1/2 2023-12-04T14:04:39.248 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-04T14:04:39.248 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:39.248 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:39.248 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-04T14:04:39.248 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:39.248 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:39.480 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-volume 2023-12-04T14:04:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7672.g7c730 12/12 2023-12-04T14:04:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7672.g7c730 1/12 2023-12-04T14:04:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-04T14:04:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-04T14:04:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-04T14:04:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-04T14:04:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-04T14:04:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-04T14:04:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-04T14:04:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-04T14:04:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-04T14:04:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-04T14:04:40.045 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-04T14:04:40.045 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:40.045 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:04:40.045 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:40.045 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-04T14:04:40.045 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-04T14:04:40.045 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-04T14:04:40.045 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-04T14:04:40.046 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-04T14:04:40.046 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-04T14:04:40.046 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-04T14:04:40.046 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-04T14:04:40.046 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-04T14:04:40.046 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-04T14:04:40.046 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-04T14:04:40.046 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:40.046 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:40.072 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:24 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:40.219 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:40.220 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:40.220 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-12-04T14:04:40.220 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:40.220 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:40.220 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 262 k 2023-12-04T14:04:40.220 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:40.220 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:40.220 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:40.220 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-12-04T14:04:40.220 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:40.220 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 262 k 2023-12-04T14:04:40.221 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 1.2 M 2023-12-04T14:04:40.221 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:40.398 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-rook 2023-12-04T14:04:40.612 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-volume-18.0.0-7672.g7c730b29.el9.noarch.rp 671 kB/s | 262 kB 00:00 2023-12-04T14:04:40.613 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:40.613 INFO:teuthology.orchestra.run.smithi146.stdout:Total 668 kB/s | 262 kB 00:00 2023-12-04T14:04:40.613 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:40.621 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:40.621 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:40.655 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:40.655 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:40.872 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:40.954 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:40.954 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-04T14:04:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:40.990 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:11 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:41.136 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:41.137 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2023-12-04T14:04:41.137 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-04T14:04:41.137 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2023-12-04T14:04:41.137 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:04:41.137 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 50 k 2023-12-04T14:04:41.137 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-04T14:04:41.138 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-04T14:04:41.138 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-04T14:04:41.138 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-04T14:04:41.138 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-04T14:04:41.138 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-04T14:04:41.138 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-04T14:04:41.138 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-04T14:04:41.138 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-04T14:04:41.138 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-04T14:04:41.138 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-04T14:04:41.139 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-04T14:04:41.139 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-04T14:04:41.139 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:41.139 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:04:41.139 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2023-12-04T14:04:41.139 INFO:teuthology.orchestra.run.smithi176.stdout:Install 13 Packages 2023-12-04T14:04:41.139 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:41.140 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 2.0 M 2023-12-04T14:04:41.140 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 26 M 2023-12-04T14:04:41.140 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:04:41.340 INFO:teuthology.orchestra.run.smithi176.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 152 kB/s | 18 kB 00:00 2023-12-04T14:04:41.365 INFO:teuthology.orchestra.run.smithi176.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 169 kB/s | 24 kB 00:00 2023-12-04T14:04:41.416 INFO:teuthology.orchestra.run.smithi176.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-12-04T14:04:41.441 INFO:teuthology.orchestra.run.smithi176.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-12-04T14:04:41.466 INFO:teuthology.orchestra.run.smithi176.stdout:(5/13): ceph-mgr-rook-18.0.0-7672.g7c730b29.el9 205 kB/s | 50 kB 00:00 2023-12-04T14:04:41.491 INFO:teuthology.orchestra.run.smithi176.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2023-12-04T14:04:41.517 INFO:teuthology.orchestra.run.smithi176.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 571 kB/s | 43 kB 00:00 2023-12-04T14:04:41.825 INFO:teuthology.orchestra.run.smithi176.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 45 kB/s | 15 kB 00:00 2023-12-04T14:04:41.859 INFO:teuthology.orchestra.run.smithi176.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 88 kB/s | 32 kB 00:00 2023-12-04T14:04:41.985 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:41.985 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:41.985 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:41.985 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:41.985 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:41.985 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:42.059 INFO:teuthology.orchestra.run.smithi176.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 364 kB/s | 197 kB 00:00 2023-12-04T14:04:42.109 INFO:teuthology.orchestra.run.smithi176.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 236 kB/s | 59 kB 00:00 2023-12-04T14:04:42.176 INFO:teuthology.orchestra.run.smithi176.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 768 kB/s | 90 kB 00:00 2023-12-04T14:04:42.321 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados-devel 2023-12-04T14:04:42.393 INFO:teuthology.orchestra.run.smithi176.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2023-12-04T14:04:42.395 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:42.395 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2023-12-04T14:04:42.396 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:04:42.409 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:04:42.409 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:04:42.509 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:04:42.511 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:04:42.909 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:26 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:43.022 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:04:43.053 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:43.054 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:43.054 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2023-12-04T14:04:43.054 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:43.054 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:43.054 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 126 k 2023-12-04T14:04:43.054 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:43.054 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:43.054 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:43.054 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-12-04T14:04:43.054 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:43.055 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 126 k 2023-12-04T14:04:43.055 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 456 k 2023-12-04T14:04:43.055 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:43.180 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-04T14:04:43.235 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-04T14:04:43.277 INFO:teuthology.orchestra.run.smithi146.stdout:librados-devel-18.0.0-7672.g7c730b29.el9.x86_64 571 kB/s | 126 kB 00:00 2023-12-04T14:04:43.277 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:43.277 INFO:teuthology.orchestra.run.smithi146.stdout:Total 567 kB/s | 126 kB 00:00 2023-12-04T14:04:43.277 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:43.283 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:43.283 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:43.291 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-04T14:04:43.328 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-04T14:04:43.338 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:43.339 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:43.369 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-04T14:04:43.471 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-04T14:04:43.483 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:43.581 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-04T14:04:43.600 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:43.637 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-04T14:04:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:44.342 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:44.342 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:44.342 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:44.343 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:44.343 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:44.343 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-04T14:04:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-04T14:04:44.443 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-04T14:04:44.491 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-04T14:04:44.513 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-rook-2:18.0.0-7672.g7c730b29.el9.noarch 13/13 2023-12-04T14:04:44.604 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs2 2023-12-04T14:04:45.193 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:29 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:45.291 INFO:teuthology.orchestra.run.smithi146.stdout:Package libcephfs2-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:04:45.328 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7672.g7c730b29.el9.noarch 13/13 2023-12-04T14:04:45.329 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7672.g7c730b29.el9.noarch 1/13 2023-12-04T14:04:45.329 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-04T14:04:45.329 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-04T14:04:45.329 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-04T14:04:45.329 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-04T14:04:45.329 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-04T14:04:45.329 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-04T14:04:45.329 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-04T14:04:45.329 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-04T14:04:45.329 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-04T14:04:45.329 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-04T14:04:45.329 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-04T14:04:45.341 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:45.342 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-12-04T14:04:45.342 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:45.406 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs-devel 2023-12-04T14:04:45.743 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-04T14:04:45.743 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:45.743 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:04:45.743 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:45.743 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-04T14:04:45.743 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-04T14:04:45.743 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-04T14:04:45.743 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-04T14:04:45.743 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-04T14:04:45.743 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-04T14:04:45.744 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-04T14:04:45.744 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-04T14:04:45.744 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-04T14:04:45.744 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-04T14:04:45.744 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-04T14:04:45.744 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-04T14:04:45.744 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:45.744 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:45.980 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:29 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:45.989 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-cephadm 2023-12-04T14:04:46.125 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 31 k 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 31 k 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 139 k 2023-12-04T14:04:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:46.447 INFO:teuthology.orchestra.run.smithi146.stdout:libcephfs-devel-18.0.0-7672.g7c730b29.el9.x86_6 97 kB/s | 31 kB 00:00 2023-12-04T14:04:46.447 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:46.447 INFO:teuthology.orchestra.run.smithi146.stdout:Total 97 kB/s | 31 kB 00:00 2023-12-04T14:04:46.447 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:46.451 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:46.451 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:46.479 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:46.479 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:46.562 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:17 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:46.566 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:46.669 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:46.708 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:46.709 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2023-12-04T14:04:46.709 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-04T14:04:46.709 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2023-12-04T14:04:46.709 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:04:46.709 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 133 k 2023-12-04T14:04:46.709 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-04T14:04:46.709 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-04T14:04:46.709 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-04T14:04:46.710 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-04T14:04:46.710 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-04T14:04:46.710 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-04T14:04:46.710 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:46.710 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:04:46.710 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2023-12-04T14:04:46.710 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2023-12-04T14:04:46.710 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:46.711 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 6.8 M 2023-12-04T14:04:46.711 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 33 M 2023-12-04T14:04:46.711 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:04:47.028 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:47.028 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-12-04T14:04:47.128 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): ceph-mgr-cephadm-18.0.0-7672.g7c730b29.e 524 kB/s | 133 kB 00:00 2023-12-04T14:04:47.204 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2023-12-04T14:04:47.291 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:47.291 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:47.292 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:47.292 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:47.292 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:47.292 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:47.462 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 174 kB/s | 58 kB 00:00 2023-12-04T14:04:47.472 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados2 2023-12-04T14:04:47.521 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 175 kB/s | 51 kB 00:00 2023-12-04T14:04:47.554 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2023-12-04T14:04:47.558 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:47.559 INFO:teuthology.orchestra.run.smithi176.stdout:Total 8.0 MB/s | 6.8 MB 00:00 2023-12-04T14:04:47.559 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:04:47.567 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:04:47.567 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:04:47.655 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:04:47.656 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:04:47.954 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:04:48.050 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:32 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-04T14:04:48.147 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:04:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-04T14:04:48.197 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:48.198 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-12-04T14:04:48.198 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:48.273 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librbd1 2023-12-04T14:04:48.688 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-04T14:04:48.776 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-04T14:04:48.837 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-04T14:04:48.851 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:32 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:48.862 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 6/6 2023-12-04T14:04:48.948 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:04:48.998 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:48.999 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-12-04T14:04:48.999 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:49.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rados 2023-12-04T14:04:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:33 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:49.648 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 6/6 2023-12-04T14:04:49.649 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 1/6 2023-12-04T14:04:49.649 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-04T14:04:49.649 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-04T14:04:49.649 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-04T14:04:49.649 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-04T14:04:49.743 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rados-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:04:49.796 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:49.797 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-12-04T14:04:49.797 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:49.869 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rgw 2023-12-04T14:04:49.970 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-04T14:04:49.970 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:49.971 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:04:49.971 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:49.971 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-04T14:04:49.971 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-04T14:04:49.971 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-04T14:04:49.971 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-04T14:04:49.971 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-04T14:04:49.971 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:49.971 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:50.221 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-fuse 2023-12-04T14:04:50.444 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:34 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:50.540 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rgw-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:04:50.590 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:50.591 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-12-04T14:04:50.591 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:50.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-cephfs 2023-12-04T14:04:50.802 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:21 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:50.948 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:50.948 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:50.948 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-04T14:04:50.948 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:50.949 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:04:50.949 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 860 k 2023-12-04T14:04:50.949 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2023-12-04T14:04:50.949 INFO:teuthology.orchestra.run.smithi176.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-04T14:04:50.949 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:50.949 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:04:50.949 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:50.949 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2023-12-04T14:04:50.949 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:50.950 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 939 k 2023-12-04T14:04:50.950 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 2.7 M 2023-12-04T14:04:50.950 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:04:51.074 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 631 kB/s | 79 kB 00:00 2023-12-04T14:04:51.224 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:35 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:51.274 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-fuse-18.0.0-7672.g7c730b29.el9.x86_ 2.6 MB/s | 860 kB 00:00 2023-12-04T14:04:51.276 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:51.276 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.8 MB/s | 939 kB 00:00 2023-12-04T14:04:51.276 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:04:51.296 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:04:51.296 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:04:51.320 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-cephfs-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:04:51.370 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:51.370 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:04:51.371 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:04:51.371 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-12-04T14:04:51.371 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:51.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rbd 2023-12-04T14:04:51.560 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:04:51.673 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-04T14:04:51.749 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 2/2 2023-12-04T14:04:52.014 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:36 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:52.108 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rbd-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:04:52.159 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:52.160 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-12-04T14:04:52.160 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:52.223 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-fuse 2023-12-04T14:04:52.430 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 2/2 2023-12-04T14:04:52.430 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 1/2 2023-12-04T14:04:52.747 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-04T14:04:52.747 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:52.747 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:04:52.747 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-04T14:04:52.747 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:52.747 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:52.800 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:36 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:52.945 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:52.946 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:52.946 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2023-12-04T14:04:52.946 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:52.946 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:52.946 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 86 k 2023-12-04T14:04:52.946 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:52.946 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:52.946 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:52.946 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-12-04T14:04:52.946 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:52.947 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 86 k 2023-12-04T14:04:52.947 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 231 k 2023-12-04T14:04:52.947 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:52.995 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-volume 2023-12-04T14:04:53.141 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-fuse-18.0.0-7672.g7c730b29.el9.x86_64.rpm 442 kB/s | 86 kB 00:00 2023-12-04T14:04:53.141 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:53.141 INFO:teuthology.orchestra.run.smithi146.stdout:Total 439 kB/s | 86 kB 00:00 2023-12-04T14:04:53.142 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:53.148 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:53.149 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:53.203 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:53.203 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:53.351 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:53.447 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:53.583 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:24 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:53.729 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:53.730 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:53.730 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2023-12-04T14:04:53.730 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:53.730 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:04:53.730 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume noarch 2:18.0.0-7672.g7c730b29.el9 ceph-noarch 262 k 2023-12-04T14:04:53.730 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:53.730 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:04:53.730 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:53.731 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-04T14:04:53.731 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:53.731 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 262 k 2023-12-04T14:04:53.731 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 1.2 M 2023-12-04T14:04:53.731 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:04:53.841 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:54.120 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-volume-18.0.0-7672.g7c730b29.el9.noarch.rp 673 kB/s | 262 kB 00:00 2023-12-04T14:04:54.121 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:54.121 INFO:teuthology.orchestra.run.smithi176.stdout:Total 671 kB/s | 262 kB 00:00 2023-12-04T14:04:54.121 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:04:54.129 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:04:54.129 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:04:54.140 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:54.140 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:54.141 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:54.141 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:54.141 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:54.141 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:54.160 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:04:54.161 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:04:54.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-mirror 2023-12-04T14:04:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:04:54.447 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-volume-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:54.462 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-volume-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:54.462 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:54.462 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-04T14:04:54.463 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:54.950 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:38 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:55.097 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:55.097 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:55.098 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2023-12-04T14:04:55.098 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:55.098 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:55.098 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 3.0 M 2023-12-04T14:04:55.098 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:55.098 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:55.098 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:55.098 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-12-04T14:04:55.098 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:55.098 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 3.0 M 2023-12-04T14:04:55.098 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 12 M 2023-12-04T14:04:55.098 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:55.552 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-volume-2:18.0.0-7672.g7c730b29.el9.noarch 1/1 2023-12-04T14:04:55.552 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:55.552 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:04:55.552 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume-2:18.0.0-7672.g7c730b29.el9.noarch 2023-12-04T14:04:55.552 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:55.552 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:55.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados-devel 2023-12-04T14:04:55.910 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-mirror-18.0.0-7672.g7c730b29.el9.x86_64.rpm 3.7 MB/s | 3.0 MB 00:00 2023-12-04T14:04:55.911 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:55.911 INFO:teuthology.orchestra.run.smithi146.stdout:Total 3.7 MB/s | 3.0 MB 00:00 2023-12-04T14:04:55.911 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:55.920 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:55.921 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:55.993 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:55.993 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:56.412 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:56.439 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:27 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:56.503 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:04:56.503 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-04T14:04:56.503 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-04T14:04:56.503 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-04T14:04:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:56.584 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:56.585 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:56.585 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2023-12-04T14:04:56.585 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:56.585 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:04:56.585 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 126 k 2023-12-04T14:04:56.585 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:56.585 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:04:56.585 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:56.586 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-04T14:04:56.586 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:56.586 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 126 k 2023-12-04T14:04:56.586 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 456 k 2023-12-04T14:04:56.586 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:04:56.933 INFO:teuthology.orchestra.run.smithi176.stdout:librados-devel-18.0.0-7672.g7c730b29.el9.x86_64 364 kB/s | 126 kB 00:00 2023-12-04T14:04:56.934 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:56.934 INFO:teuthology.orchestra.run.smithi176.stdout:Total 362 kB/s | 126 kB 00:00 2023-12-04T14:04:56.934 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:04:56.939 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:04:56.940 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:04:56.995 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:04:56.995 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:04:57.138 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:04:57.224 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:57.582 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:57.582 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:57.582 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:04:57.582 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:57.582 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:57.582 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:04:57.635 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:57.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-nbd 2023-12-04T14:04:57.928 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:57.928 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:57.928 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:04:57.928 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:04:57.928 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:57.928 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:58.173 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs2 2023-12-04T14:04:58.399 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:42 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:04:58.545 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:04:58.545 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:58.545 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2023-12-04T14:04:58.545 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:58.545 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-12-04T14:04:58.545 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 171 k 2023-12-04T14:04:58.545 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:58.545 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-12-04T14:04:58.546 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-12-04T14:04:58.546 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-12-04T14:04:58.546 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:04:58.546 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 171 k 2023-12-04T14:04:58.546 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 490 k 2023-12-04T14:04:58.546 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-12-04T14:04:58.759 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:29 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:58.858 INFO:teuthology.orchestra.run.smithi176.stdout:Package libcephfs2-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:04:58.909 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:58.910 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-04T14:04:58.910 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:04:58.975 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs-devel 2023-12-04T14:04:59.030 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-nbd-18.0.0-7672.g7c730b29.el9.x86_64.rpm 353 kB/s | 171 kB 00:00 2023-12-04T14:04:59.030 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:59.031 INFO:teuthology.orchestra.run.smithi146.stdout:Total 352 kB/s | 171 kB 00:00 2023-12-04T14:04:59.031 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-12-04T14:04:59.039 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-12-04T14:04:59.039 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-12-04T14:04:59.094 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-12-04T14:04:59.094 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-12-04T14:04:59.249 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-12-04T14:04:59.344 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:59.547 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:30 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:04:59.692 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:04:59.693 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:59.693 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2023-12-04T14:04:59.693 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:59.693 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:04:59.693 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 31 k 2023-12-04T14:04:59.693 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:59.693 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:04:59.694 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:04:59.694 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-04T14:04:59.694 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:04:59.694 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 31 k 2023-12-04T14:04:59.694 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 139 k 2023-12-04T14:04:59.694 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:04:59.814 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:04:59.928 INFO:teuthology.orchestra.run.smithi176.stdout:libcephfs-devel-18.0.0-7672.g7c730b29.el9.x86_6 133 kB/s | 31 kB 00:00 2023-12-04T14:04:59.929 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:04:59.929 INFO:teuthology.orchestra.run.smithi176.stdout:Total 132 kB/s | 31 kB 00:00 2023-12-04T14:04:59.929 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:04:59.932 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:04:59.932 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:04:59.964 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:04:59.964 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:05:00.065 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:05:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:00.107 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-12-04T14:05:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:05:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:00.107 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:05:00.169 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:00.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-mirror 2023-12-04T14:05:00.561 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:00.827 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:00.827 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:00.827 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:05:00.827 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:05:00.827 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:00.827 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:05:00.913 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:44 ago on Mon 04 Dec 2023 02:03:16 PM UTC. 2023-12-04T14:05:00.996 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados2 2023-12-04T14:05:01.016 INFO:teuthology.orchestra.run.smithi146.stdout:Package rbd-mirror-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:05:01.070 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-12-04T14:05:01.072 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-12-04T14:05:01.072 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-12-04T14:05:01.139 DEBUG:teuthology.parallel:result is None 2023-12-04T14:05:01.574 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:32 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:05:01.670 INFO:teuthology.orchestra.run.smithi176.stdout:Package librados2-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:05:01.721 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:05:01.722 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-04T14:05:01.723 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:05:01.791 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librbd1 2023-12-04T14:05:02.372 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:33 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:05:02.468 INFO:teuthology.orchestra.run.smithi176.stdout:Package librbd1-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:05:02.518 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:05:02.519 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-04T14:05:02.519 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:05:02.588 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rados 2023-12-04T14:05:03.163 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:34 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:05:03.258 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rados-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:05:03.309 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:05:03.310 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-04T14:05:03.311 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:05:03.376 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rgw 2023-12-04T14:05:03.950 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:34 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:05:04.044 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rgw-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:05:04.095 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:05:04.096 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-04T14:05:04.096 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:05:04.162 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-cephfs 2023-12-04T14:05:04.733 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:35 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:05:04.828 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-cephfs-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:05:04.878 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:05:04.879 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-04T14:05:04.879 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:05:04.945 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rbd 2023-12-04T14:05:05.519 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:36 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:05:05.613 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rbd-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:05:05.663 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:05:05.664 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-04T14:05:05.664 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:05:05.732 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-fuse 2023-12-04T14:05:06.305 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:37 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:05:06.450 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:05:06.451 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:05:06.451 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2023-12-04T14:05:06.451 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:05:06.451 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:05:06.451 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 86 k 2023-12-04T14:05:06.451 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:06.451 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:05:06.451 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:05:06.451 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-04T14:05:06.451 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:06.452 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 86 k 2023-12-04T14:05:06.452 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 231 k 2023-12-04T14:05:06.452 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:05:06.656 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-fuse-18.0.0-7672.g7c730b29.el9.x86_64.rpm 420 kB/s | 86 kB 00:00 2023-12-04T14:05:06.656 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:05:06.657 INFO:teuthology.orchestra.run.smithi176.stdout:Total 417 kB/s | 86 kB 00:00 2023-12-04T14:05:06.657 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:05:06.664 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:05:06.664 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:05:06.718 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:05:06.719 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:05:06.864 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:05:06.962 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:07.342 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:07.649 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:07.649 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:07.650 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:05:07.650 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:05:07.650 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:07.650 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:05:07.840 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-mirror 2023-12-04T14:05:08.423 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:39 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:05:08.575 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:05:08.575 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:05:08.576 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2023-12-04T14:05:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:05:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:05:08.576 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 3.0 M 2023-12-04T14:05:08.576 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:05:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:05:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-04T14:05:08.576 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 3.0 M 2023-12-04T14:05:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 12 M 2023-12-04T14:05:08.576 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:05:09.842 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-mirror-18.0.0-7672.g7c730b29.el9.x86_64.rpm 2.4 MB/s | 3.0 MB 00:01 2023-12-04T14:05:09.843 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:05:09.843 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.4 MB/s | 3.0 MB 00:01 2023-12-04T14:05:09.843 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:05:09.852 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:05:09.852 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:05:09.924 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:05:09.925 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:05:10.341 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:05:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:10.434 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T14:05:10.434 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-04T14:05:10.434 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-04T14:05:10.434 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-04T14:05:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:11.525 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:11.525 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:11.525 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:05:11.525 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:05:11.525 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:11.525 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:05:11.721 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-nbd 2023-12-04T14:05:12.314 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:43 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:05:12.460 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:05:12.460 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:05:12.460 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2023-12-04T14:05:12.460 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:05:12.461 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2023-12-04T14:05:12.461 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd x86_64 2:18.0.0-7672.g7c730b29.el9 ceph 171 k 2023-12-04T14:05:12.461 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:12.461 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2023-12-04T14:05:12.461 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2023-12-04T14:05:12.461 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2023-12-04T14:05:12.461 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:12.462 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 171 k 2023-12-04T14:05:12.462 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 490 k 2023-12-04T14:05:12.462 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2023-12-04T14:05:12.812 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-nbd-18.0.0-7672.g7c730b29.el9.x86_64.rpm 487 kB/s | 171 kB 00:00 2023-12-04T14:05:12.812 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2023-12-04T14:05:12.813 INFO:teuthology.orchestra.run.smithi176.stdout:Total 485 kB/s | 171 kB 00:00 2023-12-04T14:05:12.813 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2023-12-04T14:05:12.820 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2023-12-04T14:05:12.820 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2023-12-04T14:05:12.876 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2023-12-04T14:05:12.876 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2023-12-04T14:05:13.025 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2023-12-04T14:05:13.128 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:13.582 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-2:18.0.0-7672.g7c730b29.el9.x86_64 1/1 2023-12-04T14:05:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:13.866 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2023-12-04T14:05:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-2:18.0.0-7672.g7c730b29.el9.x86_64 2023-12-04T14:05:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:13.866 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:05:14.062 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-mirror 2023-12-04T14:05:14.672 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:45 ago on Mon 04 Dec 2023 02:03:29 PM UTC. 2023-12-04T14:05:14.774 INFO:teuthology.orchestra.run.smithi176.stdout:Package rbd-mirror-2:18.0.0-7672.g7c730b29.el9.x86_64 is already installed. 2023-12-04T14:05:14.829 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2023-12-04T14:05:14.830 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2023-12-04T14:05:14.830 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2023-12-04T14:05:14.899 DEBUG:teuthology.parallel:result is None 2023-12-04T14:05:14.899 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7c730b2996d0b00d56aea5ca791f90c534a13422 2023-12-04T14:05:15.038 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-04T14:05:15.063 INFO:teuthology.orchestra.run.smithi146.stdout:18.0.0-7672.g7c730b29.el9 2023-12-04T14:05:15.063 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7672.g7c730b29.el9 2023-12-04T14:05:15.063 INFO:teuthology.task.install:The correct ceph version 18.0.0-7672.g7c730b29 is installed. 2023-12-04T14:05:15.064 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7c730b2996d0b00d56aea5ca791f90c534a13422 2023-12-04T14:05:15.272 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-04T14:05:15.292 INFO:teuthology.orchestra.run.smithi176.stdout:18.0.0-7672.g7c730b29.el9 2023-12-04T14:05:15.292 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7672.g7c730b29.el9 2023-12-04T14:05:15.292 INFO:teuthology.task.install:The correct ceph version 18.0.0-7672.g7c730b29 is installed. 2023-12-04T14:05:15.292 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-04T14:05:15.292 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:15.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-04T14:05:15.325 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:15.325 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-04T14:05:15.358 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-04T14:05:15.359 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:15.359 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2023-12-04T14:05:15.397 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-04T14:05:15.468 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:15.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2023-12-04T14:05:15.494 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-04T14:05:15.562 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-04T14:05:15.562 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:15.562 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-04T14:05:15.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-04T14:05:15.663 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:15.663 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-04T14:05:15.691 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-04T14:05:15.758 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-04T14:05:15.758 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:15.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2023-12-04T14:05:15.788 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-04T14:05:15.858 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:15.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2023-12-04T14:05:15.885 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-04T14:05:15.952 INFO:teuthology.run_tasks:Running task ceph... 2023-12-04T14:05:16.013 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-04T14:05:16.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2023-12-04T14:05:16.016 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /var/log/ceph 2023-12-04T14:05:16.044 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-04T14:05:16.044 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-04T14:05:16.085 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-04T14:05:16.112 INFO:tasks.ceph:Creating extra log directories... 2023-12-04T14:05:16.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-04T14:05:16.154 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-04T14:05:16.193 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-12-04T14:05:16.193 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-04T14:05:16.193 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148', 'branch': 'wip-dis-testing-revert', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7477148', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing-revert'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-revert', 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2667, 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422', 'sleep_before_teardown': 0, 'subset': '17/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-revert', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422', 'targets': {'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCfbmNcyqeHhjNg3SpAV70mufKp07o/mtrEKQB1dLC4Q+M8Bfmnpw6lvzxhJ5bkx+i89G8/Yven4H30c2gnqfP8=', 'smithi176.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODo4dFnAcHFCBEVYKDs1fTrIEVFsW1LBi7z472ULX0gW0/1v17likGLMVd2GFpVeSPR9ljaeM34RM4QVXP6ObY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-04_13:47:01', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-04T14:05:16.194 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-04T14:05:16.223 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-04T14:05:16.238 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2023-12-04T14:05:16.278 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2023-12-04T14:05:16.305 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:16.306 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2023-12-04T14:05:16.361 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-04T14:05:16.361 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2023-12-04T14:05:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-04T14:05:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T14:05:16.417 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-04T14:05:16.417 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T14:05:16.417 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2023-12-04T14:05:16.417 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2023-12-04 14:04:56.727930624 +0000 2023-12-04T14:05:16.417 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2023-12-04 14:02:27.288084166 +0000 2023-12-04T14:05:16.417 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2023-12-04 14:02:27.288084166 +0000 2023-12-04T14:05:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2023-12-04 14:02:27.288084166 +0000 2023-12-04T14:05:16.418 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-04T14:05:16.483 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2023-12-04T14:05:16.484 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2023-12-04T14:05:16.484 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000204556 s, 2.5 MB/s 2023-12-04T14:05:16.485 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-04T14:05:16.542 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2023-12-04T14:05:16.597 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-04T14:05:16.597 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T14:05:16.597 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-12-04T14:05:16.597 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T14:05:16.597 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2023-12-04T14:05:16.598 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2023-12-04 14:04:56.728930609 +0000 2023-12-04T14:05:16.598 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2023-12-04 14:02:27.793076541 +0000 2023-12-04T14:05:16.598 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2023-12-04 14:02:27.793076541 +0000 2023-12-04T14:05:16.598 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2023-12-04 14:02:27.793076541 +0000 2023-12-04T14:05:16.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-04T14:05:16.663 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2023-12-04T14:05:16.663 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2023-12-04T14:05:16.663 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000213506 s, 2.4 MB/s 2023-12-04T14:05:16.664 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-04T14:05:16.720 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2023-12-04T14:05:16.775 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-04T14:05:16.775 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T14:05:16.775 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 700 Links: 1 2023-12-04T14:05:16.775 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T14:05:16.775 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2023-12-04T14:05:16.775 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2023-12-04 14:04:56.728930609 +0000 2023-12-04T14:05:16.775 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2023-12-04 14:02:28.267069386 +0000 2023-12-04T14:05:16.775 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2023-12-04 14:02:28.267069386 +0000 2023-12-04T14:05:16.775 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2023-12-04 14:02:28.267069386 +0000 2023-12-04T14:05:16.776 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-04T14:05:16.840 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2023-12-04T14:05:16.841 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2023-12-04T14:05:16.841 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000178505 s, 2.9 MB/s 2023-12-04T14:05:16.842 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-04T14:05:16.898 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2023-12-04T14:05:16.953 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-04T14:05:16.953 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T14:05:16.953 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-04T14:05:16.953 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T14:05:16.953 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2023-12-04T14:05:16.954 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2023-12-04 14:04:56.729930594 +0000 2023-12-04T14:05:16.954 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2023-12-04 14:02:28.739062260 +0000 2023-12-04T14:05:16.954 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2023-12-04 14:02:28.739062260 +0000 2023-12-04T14:05:16.954 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2023-12-04 14:02:28.739062260 +0000 2023-12-04T14:05:16.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-04T14:05:17.018 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2023-12-04T14:05:17.019 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2023-12-04T14:05:17.019 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.00019389 s, 2.6 MB/s 2023-12-04T14:05:17.020 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-04T14:05:17.076 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-04T14:05:17.076 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-04T14:05:17.076 INFO:tasks.ceph:Generating config... 2023-12-04T14:05:17.077 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-04T14:05:17.077 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-04T14:05:17.078 INFO:tasks.ceph:[client] log to stderr = False 2023-12-04T14:05:17.078 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-04T14:05:17.078 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2023-12-04T14:05:17.078 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:17.078 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:17.078 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:17.078 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:17.078 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:17.078 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:17.078 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:17.078 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:17.079 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-04T14:05:17.079 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-04T14:05:17.079 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-04T14:05:17.079 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-04T14:05:17.079 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-04T14:05:17.079 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-04T14:05:17.079 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-04T14:05:17.079 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-04T14:05:17.079 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-04T14:05:17.079 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-04T14:05:17.079 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-04T14:05:17.079 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-04T14:05:17.080 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-04T14:05:17.080 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-04T14:05:17.080 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-04T14:05:17.080 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-04T14:05:17.080 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-04T14:05:17.080 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-04T14:05:17.080 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-04T14:05:17.080 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-04T14:05:17.080 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-04T14:05:17.080 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-04T14:05:17.080 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-04T14:05:17.081 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-12-04T14:05:17.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-12-04T14:05:17.169 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/cluster1.keyring 2023-12-04T14:05:17.172 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-12-04T14:05:17.216 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-04T14:05:17.284 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.146')] 2023-12-04T14:05:17.284 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.146', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-04T14:05:17.285 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:17.285 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-04T14:05:17.340 DEBUG:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.146 --print /home/ubuntu/cephtest/cluster1.monmap 2023-12-04T14:05:17.418 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-12-04T14:05:17.418 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool: generated fsid 50899115-b691-4368-9c50-3c5017eb8b8a 2023-12-04T14:05:17.418 INFO:teuthology.orchestra.run.smithi146.stdout:setting min_mon_release = pacific 2023-12-04T14:05:17.418 INFO:teuthology.orchestra.run.smithi146.stdout:epoch 0 2023-12-04T14:05:17.419 INFO:teuthology.orchestra.run.smithi146.stdout:fsid 50899115-b691-4368-9c50-3c5017eb8b8a 2023-12-04T14:05:17.419 INFO:teuthology.orchestra.run.smithi146.stdout:last_changed 2023-12-04T14:05:17.417464+0000 2023-12-04T14:05:17.419 INFO:teuthology.orchestra.run.smithi146.stdout:created 2023-12-04T14:05:17.417464+0000 2023-12-04T14:05:17.419 INFO:teuthology.orchestra.run.smithi146.stdout:min_mon_release 16 (pacific) 2023-12-04T14:05:17.419 INFO:teuthology.orchestra.run.smithi146.stdout:election_strategy: 1 2023-12-04T14:05:17.419 INFO:teuthology.orchestra.run.smithi146.stdout:0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2023-12-04T14:05:17.419 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-12-04T14:05:17.421 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-04T14:05:17.475 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 50899115-b691-4368-9c50-3c5017eb8b8a... 2023-12-04T14:05:17.476 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-04T14:05:17.518 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-04T14:05:17.565 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2023-12-04T14:05:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2023-12-04T14:05:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-04T14:05:17.565 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2023-12-04T14:05:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2023-12-04T14:05:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2023-12-04T14:05:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2023-12-04T14:05:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2023-12-04T14:05:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.566 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2023-12-04T14:05:17.567 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2023-12-04T14:05:17.567 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.567 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-04T14:05:17.567 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2023-12-04T14:05:17.567 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.567 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2023-12-04T14:05:17.567 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-04T14:05:17.567 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.567 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2023-12-04T14:05:17.567 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.567 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2023-12-04T14:05:17.568 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2023-12-04T14:05:17.568 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2023-12-04T14:05:17.568 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2023-12-04T14:05:17.568 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2023-12-04T14:05:17.568 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2023-12-04T14:05:17.568 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2023-12-04T14:05:17.568 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-04T14:05:17.568 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-04T14:05:17.568 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2023-12-04T14:05:17.568 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.568 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-04T14:05:17.569 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.569 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2023-12-04T14:05:17.569 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.569 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2023-12-04T14:05:17.569 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.569 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2023-12-04T14:05:17.569 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2023-12-04T14:05:17.569 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2023-12-04T14:05:17.569 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = 172.21.15.146 2023-12-04T14:05:17.569 INFO:teuthology.orchestra.run.smithi146.stdout: mon client directed command retry = 5 2023-12-04T14:05:17.569 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on pool no app = False 2023-12-04T14:05:17.570 INFO:teuthology.orchestra.run.smithi146.stdout: ms inject socket failures = 5000 2023-12-04T14:05:17.570 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 50899115-b691-4368-9c50-3c5017eb8b8a 2023-12-04T14:05:17.570 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.570 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2023-12-04T14:05:17.570 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2023-12-04T14:05:17.570 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.570 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2023-12-04T14:05:17.570 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2023-12-04T14:05:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2023-12-04T14:05:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2023-12-04T14:05:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2023-12-04T14:05:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2023-12-04T14:05:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2023-12-04T14:05:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2023-12-04T14:05:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2023-12-04T14:05:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2023-12-04T14:05:17.571 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2023-12-04T14:05:17.572 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.572 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2023-12-04T14:05:17.572 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.572 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2023-12-04T14:05:17.572 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.572 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2023-12-04T14:05:17.572 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.572 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2023-12-04T14:05:17.572 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2023-12-04T14:05:17.572 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.572 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2023-12-04T14:05:17.572 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2023-12-04T14:05:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2023-12-04T14:05:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2023-12-04T14:05:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = stupid 2023-12-04T14:05:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2023-12-04T14:05:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2023-12-04T14:05:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2023-12-04T14:05:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2023-12-04T14:05:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2023-12-04T14:05:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2023-12-04T14:05:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2023-12-04T14:05:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2023-12-04T14:05:17.573 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2023-12-04T14:05:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2023-12-04T14:05:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2023-12-04T14:05:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2023-12-04T14:05:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.574 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2023-12-04T14:05:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2023-12-04T14:05:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2023-12-04T14:05:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2023-12-04T14:05:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2023-12-04T14:05:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2023-12-04T14:05:17.574 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2023-12-04T14:05:17.575 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2023-12-04T14:05:17.575 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.575 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2023-12-04T14:05:17.575 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2023-12-04T14:05:17.575 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2023-12-04T14:05:17.575 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2023-12-04T14:05:17.575 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2023-12-04T14:05:17.575 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2023-12-04T14:05:17.575 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2023-12-04T14:05:17.575 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2023-12-04T14:05:17.575 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2023-12-04T14:05:17.575 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2023-12-04T14:05:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2023-12-04T14:05:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-04T14:05:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2023-12-04T14:05:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2023-12-04T14:05:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2023-12-04T14:05:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-04T14:05:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-04T14:05:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2023-12-04T14:05:17.576 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2023-12-04T14:05:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2023-12-04T14:05:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:17.577 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2023-12-04T14:05:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2023-12-04T14:05:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2023-12-04T14:05:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2023-12-04T14:05:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-04T14:05:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-04T14:05:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: debug rbd = 20 2023-12-04T14:05:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: debug rbd_mirror = 15 2023-12-04T14:05:17.577 INFO:teuthology.orchestra.run.smithi146.stdout: log to stderr = False 2023-12-04T14:05:17.578 INFO:teuthology.orchestra.run.smithi146.stdout: rbd default features = 125 2023-12-04T14:05:17.578 INFO:teuthology.orchestra.run.smithi146.stdout: rbd mirror image policy type = none 2023-12-04T14:05:17.578 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2023-12-04T14:05:17.578 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.0] 2023-12-04T14:05:17.578 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:17.578 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:17.578 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.1] 2023-12-04T14:05:17.578 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:17.578 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:17.578 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.2] 2023-12-04T14:05:17.578 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:17.579 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:17.579 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.3] 2023-12-04T14:05:17.579 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:17.579 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:17.579 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2023-12-04T14:05:17.580 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2023-12-04T14:05:17.580 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-04T14:05:17.580 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2023-12-04T14:05:17.580 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.580 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2023-12-04T14:05:17.580 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.580 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2023-12-04T14:05:17.580 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.580 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2023-12-04T14:05:17.580 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2023-12-04T14:05:17.581 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.581 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2023-12-04T14:05:17.581 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2023-12-04T14:05:17.581 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.581 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-04T14:05:17.581 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2023-12-04T14:05:17.581 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.581 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2023-12-04T14:05:17.581 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-04T14:05:17.581 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.581 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2023-12-04T14:05:17.581 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.582 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2023-12-04T14:05:17.582 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2023-12-04T14:05:17.582 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2023-12-04T14:05:17.582 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2023-12-04T14:05:17.582 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2023-12-04T14:05:17.582 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2023-12-04T14:05:17.582 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2023-12-04T14:05:17.582 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-04T14:05:17.582 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-04T14:05:17.582 INFO:teuthology.orchestra.run.smithi176.stdout: mon_allow_pool_size_one = true 2023-12-04T14:05:17.582 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-04T14:05:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2023-12-04T14:05:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2023-12-04T14:05:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2023-12-04T14:05:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2023-12-04T14:05:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2023-12-04T14:05:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = 172.21.15.146 2023-12-04T14:05:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: mon client directed command retry = 5 2023-12-04T14:05:17.583 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on pool no app = False 2023-12-04T14:05:17.584 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject socket failures = 5000 2023-12-04T14:05:17.584 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = 50899115-b691-4368-9c50-3c5017eb8b8a 2023-12-04T14:05:17.584 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.584 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2023-12-04T14:05:17.584 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2023-12-04T14:05:17.584 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.584 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2023-12-04T14:05:17.584 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2023-12-04T14:05:17.584 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock profile = high_recovery_ops 2023-12-04T14:05:17.584 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.584 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2023-12-04T14:05:17.584 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug trim objects = true 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2023-12-04T14:05:17.585 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.586 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2023-12-04T14:05:17.586 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.586 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2023-12-04T14:05:17.586 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2023-12-04T14:05:17.586 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.586 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2023-12-04T14:05:17.586 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2023-12-04T14:05:17.586 INFO:teuthology.orchestra.run.smithi176.stdout: bdev async discard = True 2023-12-04T14:05:17.586 INFO:teuthology.orchestra.run.smithi176.stdout: bdev enable discard = True 2023-12-04T14:05:17.586 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore allocator = stupid 2023-12-04T14:05:17.586 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2023-12-04T14:05:17.586 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2023-12-04T14:05:17.587 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 1/20 2023-12-04T14:05:17.587 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 1/20 2023-12-04T14:05:17.587 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2023-12-04T14:05:17.587 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2023-12-04T14:05:17.587 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 4/10 2023-12-04T14:05:17.587 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2023-12-04T14:05:17.587 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2023-12-04T14:05:17.587 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2023-12-04T14:05:17.587 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2023-12-04T14:05:17.587 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2023-12-04T14:05:17.587 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.587 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2023-12-04T14:05:17.588 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2023-12-04T14:05:17.589 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-04T14:05:17.590 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-04T14:05:17.590 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.590 INFO:teuthology.orchestra.run.smithi176.stdout: # 1m isn't quite enough 2023-12-04T14:05:17.590 INFO:teuthology.orchestra.run.smithi176.stdout: mon_down_mkfs_grace = 2m 2023-12-04T14:05:17.590 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.590 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_filestore_osds = false 2023-12-04T14:05:17.590 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:17.590 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2023-12-04T14:05:17.590 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2023-12-04T14:05:17.590 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2023-12-04T14:05:17.590 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2023-12-04T14:05:17.591 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-04T14:05:17.591 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-04T14:05:17.591 INFO:teuthology.orchestra.run.smithi176.stdout: debug rbd = 20 2023-12-04T14:05:17.591 INFO:teuthology.orchestra.run.smithi176.stdout: debug rbd_mirror = 15 2023-12-04T14:05:17.591 INFO:teuthology.orchestra.run.smithi176.stdout: log to stderr = False 2023-12-04T14:05:17.591 INFO:teuthology.orchestra.run.smithi176.stdout: rbd default features = 125 2023-12-04T14:05:17.591 INFO:teuthology.orchestra.run.smithi176.stdout: rbd mirror image policy type = none 2023-12-04T14:05:17.591 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2023-12-04T14:05:17.591 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.0] 2023-12-04T14:05:17.591 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:17.591 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:17.592 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.1] 2023-12-04T14:05:17.592 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:17.592 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:17.592 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.2] 2023-12-04T14:05:17.592 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:17.592 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:17.592 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.3] 2023-12-04T14:05:17.592 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:17.592 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:17.594 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-12-04T14:05:17.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-12-04T14:05:17.689 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-04T14:05:17.689 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:17.689 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-12-04T14:05:17.704 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:17.704 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-12-04T14:05:17.759 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2023-12-04T14:05:17.759 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:17.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-04T14:05:17.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-04T14:05:17.838 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:17.839 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-04T14:05:17.893 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2023-12-04T14:05:17.893 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:17.893 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-04T14:05:17.893 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-04T14:05:17.939 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:17.939 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-04T14:05:17.995 INFO:tasks.ceph:Setting up mon nodes... 2023-12-04T14:05:17.995 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-04T14:05:17.996 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-04T14:05:18.050 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-04T14:05:18.052 INFO:tasks.ceph:Setting up mds nodes... 2023-12-04T14:05:18.053 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-04T14:05:18.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-12-04T14:05:18.095 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-12-04T14:05:18.110 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-12-04T14:05:18.166 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-12-04T14:05:18.182 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-04T14:05:18.226 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-04T14:05:18.242 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-04T14:05:18.284 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-04T14:05:18.299 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-04T14:05:18.381 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-04T14:05:18.396 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-04T14:05:18.478 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-04T14:05:18.493 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-04T14:05:18.576 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-04T14:05:18.591 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-04T14:05:18.673 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-04T14:05:18.688 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-04T14:05:18.771 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-04T14:05:18.786 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-04T14:05:18.786 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-04T14:05:18.786 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-12-04T14:05:18.814 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-04T14:05:18.814 INFO:tasks.ceph:role: cluster1.osd.0 2023-12-04T14:05:18.815 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com 2023-12-04T14:05:18.815 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-04T14:05:18.889 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T14:05:18.889 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T14:05:18.889 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T14:05:18.889 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-04T14:05:18.889 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T14:05:18.889 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2023-12-04T14:05:18.889 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T14:05:18.889 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-04T14:05:18.889 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T14:05:18.889 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T14:05:19.374 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2023-12-04T14:05:19.376 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2023-12-04T14:05:19.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-12-04T14:05:19.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-12-04T14:05:19.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-12-04T14:05:19.617 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-04T14:05:19.617 INFO:tasks.ceph:role: cluster1.osd.1 2023-12-04T14:05:19.618 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com 2023-12-04T14:05:19.618 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-04T14:05:19.694 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T14:05:19.694 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T14:05:19.694 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T14:05:19.695 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-04T14:05:19.695 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T14:05:19.695 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2023-12-04T14:05:19.695 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T14:05:19.695 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-04T14:05:19.695 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T14:05:19.695 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T14:05:20.149 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2023-12-04T14:05:20.151 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2023-12-04T14:05:20.152 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-12-04T14:05:20.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-12-04T14:05:20.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-12-04T14:05:20.337 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-04T14:05:20.337 INFO:tasks.ceph:role: cluster1.osd.2 2023-12-04T14:05:20.337 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com 2023-12-04T14:05:20.337 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-04T14:05:20.412 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T14:05:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T14:05:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T14:05:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-04T14:05:20.412 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T14:05:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2023-12-04T14:05:20.413 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T14:05:20.413 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-04T14:05:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T14:05:20.413 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T14:05:20.896 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2023-12-04T14:05:20.898 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2023-12-04T14:05:20.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-12-04T14:05:21.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-12-04T14:05:21.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-04T14:05:21.180 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:21.178+0000 7ff218e12500 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-12-04T14:05:21.181 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:21.179+0000 7ff218e12500 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-12-04T14:05:21.181 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:21.179+0000 7ff218e12500 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-04T14:05:21.181 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:21.179+0000 7ff218e12500 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-04T14:05:21.181 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:21.179+0000 7ff218e12500 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-04T14:05:21.188 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:21.186+0000 7ff218e12500 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-12-04T14:05:24.037 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-12-04T14:05:24.068 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-04T14:05:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:24.157+0000 7f7a846ab500 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-12-04T14:05:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:24.157+0000 7f7a846ab500 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-12-04T14:05:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:24.157+0000 7f7a846ab500 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-04T14:05:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:24.157+0000 7f7a846ab500 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-04T14:05:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:24.157+0000 7f7a846ab500 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-04T14:05:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:24.158+0000 7f7a846ab500 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-12-04T14:05:27.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-12-04T14:05:27.045 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-04T14:05:27.136 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:27.134+0000 7ff463e11500 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-12-04T14:05:27.136 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:27.134+0000 7ff463e11500 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-12-04T14:05:27.137 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:27.135+0000 7ff463e11500 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-04T14:05:27.137 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:27.135+0000 7ff463e11500 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-04T14:05:27.137 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:27.135+0000 7ff463e11500 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-04T14:05:27.137 INFO:teuthology.orchestra.run.smithi146.stderr:2023-12-04T14:05:27.135+0000 7ff463e11500 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-12-04T14:05:29.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-12-04T14:05:30.026 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-04T14:05:30.026 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:30.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-12-04T14:05:30.096 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:30.096 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-12-04T14:05:30.165 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:30.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-12-04T14:05:30.233 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:30.233 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-12-04T14:05:30.301 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:30.301 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-12-04T14:05:30.355 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:30.356 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-12-04T14:05:30.374 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:30.374 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-12-04T14:05:30.432 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:30.432 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-12-04T14:05:30.490 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:30.490 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-12-04T14:05:30.546 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:30.546 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-12-04T14:05:30.603 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:30.603 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-12-04T14:05:30.660 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:30.660 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-12-04T14:05:30.717 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:30.717 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-12-04T14:05:30.772 INFO:tasks.ceph:Adding keys to all mons... 2023-12-04T14:05:30.773 DEBUG:teuthology.orchestra.run.smithi146:> sudo tee -a /etc/ceph/cluster1.keyring 2023-12-04T14:05:30.799 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.x] 2023-12-04T14:05:30.799 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCe3G1lhlv7AhAAFQR160MPZC3SifCfYUWoGA== 2023-12-04T14:05:30.800 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2023-12-04T14:05:30.800 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCh3G1lAya1ChAAjSMa8W0D0G+DQCa0GaEtYA== 2023-12-04T14:05:30.800 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2023-12-04T14:05:30.800 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCk3G1lRfhtCRAAvSxDUZuchPx5pNTEyIKevg== 2023-12-04T14:05:30.800 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2023-12-04T14:05:30.800 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCn3G1lHHUXCBAA0eCubkJ4HfWZEY/PJR34lA== 2023-12-04T14:05:30.800 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2023-12-04T14:05:30.800 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCe3G1l50SlBRAA7jwGsaQDqzKUPdLsIquRlw== 2023-12-04T14:05:30.800 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror] 2023-12-04T14:05:30.800 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCe3G1lzO7eCRAAqWjJXldeL0QRQcvlJOhL4g== 2023-12-04T14:05:30.800 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.0] 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCe3G1lqNZ/DRAANcSohWEy5uZMHA+P7AtxxQ== 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.1] 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCe3G1lpgXxEBAAffDZKuvnVR1SZBA/+x4K2w== 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.2] 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCe3G1lZg63FhAA5pNb4Gzqc4UFjkMqWXYM3A== 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.3] 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCe3G1lrNuCHBAA2FJPniiU5Peh7A1lC8H3ug== 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.4] 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCe3G1lpaNTIhAANACQe359bpngCUCyu0YDqA== 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.5] 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCe3G1lB5ofKBAAa08+U3fB5nxZs0EIX0rv1w== 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.6] 2023-12-04T14:05:30.801 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCe3G1leML0LRAAeoD66GQpAs3mLxFQyw6PYA== 2023-12-04T14:05:30.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-04T14:05:30.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T14:05:30.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T14:05:31.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T14:05:31.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:05:31.106 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:05:31.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:05:31.215 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:05:31.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:05:31.323 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:05:31.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:05:31.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:05:31.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:05:31.541 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-04T14:05:31.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-12-04T14:05:31.567 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-12-04T14:05:31.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-12-04T14:05:31.690 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-12-04T14:05:31.744 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-12-04T14:05:31.745 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-12-04T14:05:31.745 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-12-04T14:05:31.786 INFO:tasks.ceph.cluster1.mon.a:Started 2023-12-04T14:05:31.786 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-12-04T14:05:31.787 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-12-04T14:05:31.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-12-04T14:05:31.788 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-12-04T14:05:31.788 INFO:tasks.ceph:Setting crush tunables to default 2023-12-04T14:05:31.789 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster cluster1 osd crush tunables default 2023-12-04T14:05:31.965 INFO:teuthology.orchestra.run.smithi146.stderr:adjusted tunables profile to default 2023-12-04T14:05:31.976 INFO:tasks.ceph:check_enable_crimson: False 2023-12-04T14:05:31.977 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-12-04T14:05:31.977 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:31.977 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-12-04T14:05:32.003 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:32.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-12-04T14:05:32.072 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:32.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-12-04T14:05:32.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster cluster1 osd new 59fa9f13-b138-45e2-bd6d-be19b9f474a9 0 2023-12-04T14:05:32.360 INFO:teuthology.orchestra.run.smithi146.stdout:0 2023-12-04T14:05:32.372 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster cluster1 osd new 04534fec-04a9-4ef5-8e3c-06425b73d1bf 1 2023-12-04T14:05:32.546 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:32.544+0000 7f084a832040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-04T14:05:32.552 INFO:teuthology.orchestra.run.smithi146.stdout:1 2023-12-04T14:05:32.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster cluster1 osd new 35b1d109-4378-4ebd-8d8c-b3f2eba7012b 2 2023-12-04T14:05:32.601 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:32.600+0000 7f084a832040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-04T14:05:32.741 INFO:teuthology.orchestra.run.smithi146.stdout:2 2023-12-04T14:05:32.753 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-12-04T14:05:32.753 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-12-04T14:05:32.795 INFO:tasks.ceph.cluster1.osd.0:Started 2023-12-04T14:05:32.795 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-12-04T14:05:32.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-12-04T14:05:32.797 INFO:tasks.ceph.cluster1.osd.1:Started 2023-12-04T14:05:32.797 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-12-04T14:05:32.797 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-12-04T14:05:32.799 INFO:tasks.ceph.cluster1.osd.2:Started 2023-12-04T14:05:32.800 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-04T14:05:32.995 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:32.995 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":5,"fsid":"50899115-b691-4368-9c50-3c5017eb8b8a","created":"2023-12-04T14:05:31.886833+0000","modified":"2023-12-04T14:05:32.740700+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T14:05:32.740700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"59fa9f13-b138-45e2-bd6d-be19b9f474a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"04534fec-04a9-4ef5-8e3c-06425b73d1bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"35b1d109-4378-4ebd-8d8c-b3f2eba7012b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T14:05:33.007 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-12-04T14:05:33.007 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-04T14:05:33.161 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:33.159+0000 7f084a832040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-04T14:05:33.214 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:33.213+0000 7f084a832040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-04T14:05:33.258 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:33.257+0000 7f084a832040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-04T14:05:33.309 DEBUG:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-04T14:05:33.312 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:33.310+0000 7f084a832040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-04T14:05:33.356 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:33.354+0000 7f084a832040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-04T14:05:33.397 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:05:33.396+0000 7f78ca7f9500 -1 Falling back to public interface 2023-12-04T14:05:33.401 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:05:33.399+0000 7f519561f500 -1 Falling back to public interface 2023-12-04T14:05:33.442 INFO:tasks.ceph.cluster1.osd.1.smithi146.stderr:2023-12-04T14:05:33.440+0000 7f0c36eab500 -1 Falling back to public interface 2023-12-04T14:05:33.466 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:33.464+0000 7f084a832040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-04T14:05:33.470 INFO:teuthology.misc.health.smithi146.stdout: 2023-12-04T14:05:33.470 INFO:teuthology.misc.health.smithi146.stdout:{"epoch":5,"fsid":"50899115-b691-4368-9c50-3c5017eb8b8a","created":"2023-12-04T14:05:31.886833+0000","modified":"2023-12-04T14:05:32.740700+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T14:05:32.740700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"59fa9f13-b138-45e2-bd6d-be19b9f474a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"04534fec-04a9-4ef5-8e3c-06425b73d1bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"35b1d109-4378-4ebd-8d8c-b3f2eba7012b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T14:05:33.480 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-04T14:05:33.537 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:33.536+0000 7f084a832040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-04T14:05:33.582 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:33.581+0000 7f084a832040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-04T14:05:33.634 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:33.633+0000 7f084a832040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-04T14:05:33.682 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:33.681+0000 7f084a832040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-04T14:05:34.069 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:34.067+0000 7f084a832040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-04T14:05:34.203 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:34.201+0000 7f084a832040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-04T14:05:34.335 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:34.333+0000 7f084a832040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-04T14:05:34.381 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:34.379+0000 7f084a832040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-04T14:05:34.477 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:34.476+0000 7f084a832040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-04T14:05:34.574 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:34.572+0000 7f084a832040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-04T14:05:34.712 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:34.711+0000 7f084a832040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-04T14:05:34.754 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:34.752+0000 7f084a832040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-04T14:05:34.897 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:34.895+0000 7f084a832040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-04T14:05:35.700 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:35.698+0000 7f084a832040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-04T14:05:35.976 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:35.974+0000 7f084a832040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-04T14:05:36.026 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:36.024+0000 7f084a832040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-04T14:05:36.125 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:36.123+0000 7f084a832040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-04T14:05:36.936 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:05:36.934+0000 7f78ca7f9500 -1 osd.0 0 log_to_monitors true 2023-12-04T14:05:36.944 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:05:36.942+0000 7f519561f500 -1 osd.2 0 log_to_monitors true 2023-12-04T14:05:36.983 INFO:tasks.ceph.cluster1.osd.1.smithi146.stderr:2023-12-04T14:05:36.981+0000 7f0c36eab500 -1 osd.1 0 log_to_monitors true 2023-12-04T14:05:37.397 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:05:37.395+0000 7f07eb4ab640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-04T14:05:39.783 DEBUG:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-04T14:05:40.180 INFO:teuthology.misc.health.smithi146.stdout: 2023-12-04T14:05:40.180 INFO:teuthology.misc.health.smithi146.stdout:{"epoch":7,"fsid":"50899115-b691-4368-9c50-3c5017eb8b8a","created":"2023-12-04T14:05:31.886833+0000","modified":"2023-12-04T14:05:38.872702+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T14:05:32.740700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"59fa9f13-b138-45e2-bd6d-be19b9f474a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"04534fec-04a9-4ef5-8e3c-06425b73d1bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"35b1d109-4378-4ebd-8d8c-b3f2eba7012b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T14:05:40.195 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-04T14:05:40.660 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:05:40.659+0000 7f5190740640 -1 osd.2 0 waiting for initial osdmap 2023-12-04T14:05:40.676 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:05:40.675+0000 7f5188d73640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T14:05:40.747 INFO:tasks.ceph.cluster1.osd.1.smithi146.stderr:2023-12-04T14:05:40.746+0000 7f0c32614640 -1 osd.1 0 waiting for initial osdmap 2023-12-04T14:05:40.754 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:05:40.753+0000 7f78c779c640 -1 osd.0 0 waiting for initial osdmap 2023-12-04T14:05:40.767 INFO:tasks.ceph.cluster1.osd.1.smithi146.stderr:2023-12-04T14:05:40.766+0000 7f0c2ac47640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T14:05:40.770 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:05:40.769+0000 7f78be595640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T14:05:46.498 DEBUG:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-04T14:05:46.777 INFO:teuthology.misc.health.smithi146.stdout: 2023-12-04T14:05:46.777 INFO:teuthology.misc.health.smithi146.stdout:{"epoch":11,"fsid":"50899115-b691-4368-9c50-3c5017eb8b8a","created":"2023-12-04T14:05:31.886833+0000","modified":"2023-12-04T14:05:44.888357+0000","last_up_change":"2023-12-04T14:05:41.661101+0000","last_in_change":"2023-12-04T14:05:32.740700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T14:05:42.380562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"59fa9f13-b138-45e2-bd6d-be19b9f474a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":38073366},{"type":"v1","addr":"172.21.15.146:6801","nonce":38073366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":38073366},{"type":"v1","addr":"172.21.15.146:6803","nonce":38073366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":38073366},{"type":"v1","addr":"172.21.15.146:6807","nonce":38073366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":38073366},{"type":"v1","addr":"172.21.15.146:6805","nonce":38073366}]},"public_addr":"172.21.15.146:6801/38073366","cluster_addr":"172.21.15.146:6803/38073366","heartbeat_back_addr":"172.21.15.146:6807/38073366","heartbeat_front_addr":"172.21.15.146:6805/38073366","state":["exists","up"]},{"osd":1,"uuid":"04534fec-04a9-4ef5-8e3c-06425b73d1bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":2125278788},{"type":"v1","addr":"172.21.15.146:6817","nonce":2125278788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":2125278788},{"type":"v1","addr":"172.21.15.146:6819","nonce":2125278788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":2125278788},{"type":"v1","addr":"172.21.15.146:6823","nonce":2125278788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":2125278788},{"type":"v1","addr":"172.21.15.146:6821","nonce":2125278788}]},"public_addr":"172.21.15.146:6817/2125278788","cluster_addr":"172.21.15.146:6819/2125278788","heartbeat_back_addr":"172.21.15.146:6823/2125278788","heartbeat_front_addr":"172.21.15.146:6821/2125278788","state":["exists","up"]},{"osd":2,"uuid":"35b1d109-4378-4ebd-8d8c-b3f2eba7012b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2401807565},{"type":"v1","addr":"172.21.15.146:6809","nonce":2401807565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2401807565},{"type":"v1","addr":"172.21.15.146:6811","nonce":2401807565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2401807565},{"type":"v1","addr":"172.21.15.146:6815","nonce":2401807565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2401807565},{"type":"v1","addr":"172.21.15.146:6813","nonce":2401807565}]},"public_addr":"172.21.15.146:6809/2401807565","cluster_addr":"172.21.15.146:6811/2401807565","heartbeat_back_addr":"172.21.15.146:6815/2401807565","heartbeat_front_addr":"172.21.15.146:6813/2401807565","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:05:37.973715+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:05:37.985421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:05:37.939636+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T14:05:46.788 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-04T14:05:46.788 INFO:tasks.ceph:Creating RBD pool 2023-12-04T14:05:46.789 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-12-04T14:05:47.896 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'rbd' created 2023-12-04T14:05:47.914 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-04T14:05:48.903 INFO:teuthology.orchestra.run.smithi146.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-04T14:05:48.920 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-12-04T14:05:48.921 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-12-04T14:05:48.921 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-12-04T14:05:48.921 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-12-04T14:05:48.922 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-04T14:05:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:49.259 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1670433710},{"type":"v1","addr":"172.21.15.146:6825","nonce":1670433710}]},"active_addr":"172.21.15.146:6825/1670433710","active_change":"2023-12-04T14:05:36.136782+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":894185169}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":1113888747}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":2983672918}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":1436981984}]}]} 2023-12-04T14:05:49.261 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-12-04T14:05:49.261 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-12-04T14:05:49.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-04T14:05:49.540 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":13,"fsid":"50899115-b691-4368-9c50-3c5017eb8b8a","created":"2023-12-04T14:05:31.886833+0000","modified":"2023-12-04T14:05:48.899763+0000","last_up_change":"2023-12-04T14:05:41.661101+0000","last_in_change":"2023-12-04T14:05:32.740700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T14:05:42.380562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-04T14:05:47.088489+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"59fa9f13-b138-45e2-bd6d-be19b9f474a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":38073366},{"type":"v1","addr":"172.21.15.146:6801","nonce":38073366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":38073366},{"type":"v1","addr":"172.21.15.146:6803","nonce":38073366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":38073366},{"type":"v1","addr":"172.21.15.146:6807","nonce":38073366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":38073366},{"type":"v1","addr":"172.21.15.146:6805","nonce":38073366}]},"public_addr":"172.21.15.146:6801/38073366","cluster_addr":"172.21.15.146:6803/38073366","heartbeat_back_addr":"172.21.15.146:6807/38073366","heartbeat_front_addr":"172.21.15.146:6805/38073366","state":["exists","up"]},{"osd":1,"uuid":"04534fec-04a9-4ef5-8e3c-06425b73d1bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":2125278788},{"type":"v1","addr":"172.21.15.146:6817","nonce":2125278788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":2125278788},{"type":"v1","addr":"172.21.15.146:6819","nonce":2125278788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":2125278788},{"type":"v1","addr":"172.21.15.146:6823","nonce":2125278788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":2125278788},{"type":"v1","addr":"172.21.15.146:6821","nonce":2125278788}]},"public_addr":"172.21.15.146:6817/2125278788","cluster_addr":"172.21.15.146:6819/2125278788","heartbeat_back_addr":"172.21.15.146:6823/2125278788","heartbeat_front_addr":"172.21.15.146:6821/2125278788","state":["exists","up"]},{"osd":2,"uuid":"35b1d109-4378-4ebd-8d8c-b3f2eba7012b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2401807565},{"type":"v1","addr":"172.21.15.146:6809","nonce":2401807565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2401807565},{"type":"v1","addr":"172.21.15.146:6811","nonce":2401807565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2401807565},{"type":"v1","addr":"172.21.15.146:6815","nonce":2401807565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2401807565},{"type":"v1","addr":"172.21.15.146:6813","nonce":2401807565}]},"public_addr":"172.21.15.146:6809/2401807565","cluster_addr":"172.21.15.146:6811/2401807565","heartbeat_back_addr":"172.21.15.146:6815/2401807565","heartbeat_front_addr":"172.21.15.146:6813/2401807565","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:05:37.973715+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:05:37.985421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:05:37.939636+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T14:05:49.553 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-12-04T14:05:49.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-04T14:05:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:49.833 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":13,"fsid":"50899115-b691-4368-9c50-3c5017eb8b8a","created":"2023-12-04T14:05:31.886833+0000","modified":"2023-12-04T14:05:48.899763+0000","last_up_change":"2023-12-04T14:05:41.661101+0000","last_in_change":"2023-12-04T14:05:32.740700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T14:05:42.380562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-04T14:05:47.088489+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"59fa9f13-b138-45e2-bd6d-be19b9f474a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":38073366},{"type":"v1","addr":"172.21.15.146:6801","nonce":38073366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":38073366},{"type":"v1","addr":"172.21.15.146:6803","nonce":38073366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":38073366},{"type":"v1","addr":"172.21.15.146:6807","nonce":38073366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":38073366},{"type":"v1","addr":"172.21.15.146:6805","nonce":38073366}]},"public_addr":"172.21.15.146:6801/38073366","cluster_addr":"172.21.15.146:6803/38073366","heartbeat_back_addr":"172.21.15.146:6807/38073366","heartbeat_front_addr":"172.21.15.146:6805/38073366","state":["exists","up"]},{"osd":1,"uuid":"04534fec-04a9-4ef5-8e3c-06425b73d1bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":2125278788},{"type":"v1","addr":"172.21.15.146:6817","nonce":2125278788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":2125278788},{"type":"v1","addr":"172.21.15.146:6819","nonce":2125278788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":2125278788},{"type":"v1","addr":"172.21.15.146:6823","nonce":2125278788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":2125278788},{"type":"v1","addr":"172.21.15.146:6821","nonce":2125278788}]},"public_addr":"172.21.15.146:6817/2125278788","cluster_addr":"172.21.15.146:6819/2125278788","heartbeat_back_addr":"172.21.15.146:6823/2125278788","heartbeat_front_addr":"172.21.15.146:6821/2125278788","state":["exists","up"]},{"osd":2,"uuid":"35b1d109-4378-4ebd-8d8c-b3f2eba7012b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2401807565},{"type":"v1","addr":"172.21.15.146:6809","nonce":2401807565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2401807565},{"type":"v1","addr":"172.21.15.146:6811","nonce":2401807565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2401807565},{"type":"v1","addr":"172.21.15.146:6815","nonce":2401807565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2401807565},{"type":"v1","addr":"172.21.15.146:6813","nonce":2401807565}]},"public_addr":"172.21.15.146:6809/2401807565","cluster_addr":"172.21.15.146:6811/2401807565","heartbeat_back_addr":"172.21.15.146:6815/2401807565","heartbeat_front_addr":"172.21.15.146:6813/2401807565","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:05:37.973715+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:05:37.985421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:05:37.939636+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T14:05:49.846 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-12-04T14:05:49.846 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-12-04T14:05:49.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-12-04T14:05:49.987 INFO:teuthology.orchestra.run.smithi146.stdout:34359738374 2023-12-04T14:05:49.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-04T14:05:50.008 INFO:teuthology.orchestra.run.smithi146.stdout:34359738373 2023-12-04T14:05:50.008 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-04T14:05:50.009 INFO:teuthology.orchestra.run.smithi146.stdout:34359738373 2023-12-04T14:05:50.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-04T14:05:50.299 INFO:teuthology.orchestra.run.smithi146.stdout:34359738373 2023-12-04T14:05:50.312 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.2 2023-12-04T14:05:50.345 INFO:teuthology.orchestra.run.smithi146.stdout:34359738372 2023-12-04T14:05:50.358 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2023-12-04T14:05:50.358 INFO:teuthology.orchestra.run.smithi146.stdout:34359738372 2023-12-04T14:05:50.371 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2023-12-04T14:05:51.313 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-04T14:05:51.359 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-04T14:05:51.372 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-04T14:05:51.627 INFO:teuthology.orchestra.run.smithi146.stdout:34359738374 2023-12-04T14:05:51.640 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.2 2023-12-04T14:05:51.640 DEBUG:teuthology.parallel:result is None 2023-12-04T14:05:51.654 INFO:teuthology.orchestra.run.smithi146.stdout:34359738373 2023-12-04T14:05:51.667 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.0 2023-12-04T14:05:51.667 DEBUG:teuthology.parallel:result is None 2023-12-04T14:05:51.708 INFO:teuthology.orchestra.run.smithi146.stdout:34359738373 2023-12-04T14:05:51.721 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2023-12-04T14:05:51.721 DEBUG:teuthology.parallel:result is None 2023-12-04T14:05:51.721 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-04T14:05:51.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-04T14:05:52.002 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:52.003 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2023-12-04T14:05:52.014 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-12-04T14:05:50.142326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82568,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032952,"statfs":{"total":289910292480,"available":289825742848,"internally_reserved":0,"allocated":1531904,"data_stored":1274729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.245942"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:49.909082+0000","last_change":"2023-12-04T14:05:48.912820+0000","last_active":"2023-12-04T14:05:49.909082+0000","last_peered":"2023-12-04T14:05:49.909082+0000","last_clean":"2023-12-04T14:05:49.909082+0000","last_became_active":"2023-12-04T14:05:48.912549+0000","last_became_peered":"2023-12-04T14:05:48.912549+0000","last_unstale":"2023-12-04T14:05:49.909082+0000","last_undegraded":"2023-12-04T14:05:49.909082+0000","last_fullsized":"2023-12-04T14:05:49.909082+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:32:03.090571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:49.909040+0000","last_change":"2023-12-04T14:05:48.912399+0000","last_active":"2023-12-04T14:05:49.909040+0000","last_peered":"2023-12-04T14:05:49.909040+0000","last_clean":"2023-12-04T14:05:49.909040+0000","last_became_active":"2023-12-04T14:05:48.912158+0000","last_became_peered":"2023-12-04T14:05:48.912158+0000","last_unstale":"2023-12-04T14:05:49.909040+0000","last_undegraded":"2023-12-04T14:05:49.909040+0000","last_fullsized":"2023-12-04T14:05:49.909040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T23:21:47.742096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:49.908976+0000","last_change":"2023-12-04T14:05:48.911794+0000","last_active":"2023-12-04T14:05:49.908976+0000","last_peered":"2023-12-04T14:05:49.908976+0000","last_clean":"2023-12-04T14:05:49.908976+0000","last_became_active":"2023-12-04T14:05:48.911550+0000","last_became_peered":"2023-12-04T14:05:48.911550+0000","last_unstale":"2023-12-04T14:05:49.908976+0000","last_undegraded":"2023-12-04T14:05:49.908976+0000","last_fullsized":"2023-12-04T14:05:49.908976+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:22:50.543992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:49.909088+0000","last_change":"2023-12-04T14:05:48.912242+0000","last_active":"2023-12-04T14:05:49.909088+0000","last_peered":"2023-12-04T14:05:49.909088+0000","last_clean":"2023-12-04T14:05:49.909088+0000","last_became_active":"2023-12-04T14:05:48.911963+0000","last_became_peered":"2023-12-04T14:05:48.911963+0000","last_unstale":"2023-12-04T14:05:49.909088+0000","last_undegraded":"2023-12-04T14:05:49.909088+0000","last_fullsized":"2023-12-04T14:05:49.909088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T23:00:55.264982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-04T14:05:48.911963+0000","last_change":"2023-12-04T14:05:48.910743+0000","last_active":"2023-12-04T14:05:48.911963+0000","last_peered":"2023-12-04T14:05:48.911963+0000","last_clean":"2023-12-04T14:05:48.911963+0000","last_became_active":"2023-12-04T14:05:48.910461+0000","last_became_peered":"2023-12-04T14:05:48.910461+0000","last_unstale":"2023-12-04T14:05:48.911963+0000","last_undegraded":"2023-12-04T14:05:48.911963+0000","last_fullsized":"2023-12-04T14:05:48.911963+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:05:52.249003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-04T14:05:48.913414+0000","last_change":"2023-12-04T14:05:48.911076+0000","last_active":"2023-12-04T14:05:48.913414+0000","last_peered":"2023-12-04T14:05:48.913414+0000","last_clean":"2023-12-04T14:05:48.913414+0000","last_became_active":"2023-12-04T14:05:48.910755+0000","last_became_peered":"2023-12-04T14:05:48.910755+0000","last_unstale":"2023-12-04T14:05:48.913414+0000","last_undegraded":"2023-12-04T14:05:48.913414+0000","last_fullsized":"2023-12-04T14:05:48.913414+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:08:14.562280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-04T14:05:48.913437+0000","last_change":"2023-12-04T14:05:48.911361+0000","last_active":"2023-12-04T14:05:48.913437+0000","last_peered":"2023-12-04T14:05:48.913437+0000","last_clean":"2023-12-04T14:05:48.913437+0000","last_became_active":"2023-12-04T14:05:48.910748+0000","last_became_peered":"2023-12-04T14:05:48.910748+0000","last_unstale":"2023-12-04T14:05:48.913437+0000","last_undegraded":"2023-12-04T14:05:48.913437+0000","last_fullsized":"2023-12-04T14:05:48.913437+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:54:02.615361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:49.908936+0000","last_change":"2023-12-04T14:05:48.911395+0000","last_active":"2023-12-04T14:05:49.908936+0000","last_peered":"2023-12-04T14:05:49.908936+0000","last_clean":"2023-12-04T14:05:49.908936+0000","last_became_active":"2023-12-04T14:05:48.911100+0000","last_became_peered":"2023-12-04T14:05:48.911100+0000","last_unstale":"2023-12-04T14:05:49.908936+0000","last_undegraded":"2023-12-04T14:05:49.908936+0000","last_fullsized":"2023-12-04T14:05:49.908936+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T00:43:32.547721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-04T14:05:48.905100+0000","last_change":"2023-12-04T14:05:43.891085+0000","last_active":"2023-12-04T14:05:48.905100+0000","last_peered":"2023-12-04T14:05:48.905100+0000","last_clean":"2023-12-04T14:05:48.905100+0000","last_became_active":"2023-12-04T14:05:43.890739+0000","last_became_peered":"2023-12-04T14:05:43.890739+0000","last_unstale":"2023-12-04T14:05:48.905100+0000","last_undegraded":"2023-12-04T14:05:48.905100+0000","last_fullsized":"2023-12-04T14:05:48.905100+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:42.882931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:42.882931+0000","last_clean_scrub_stamp":"2023-12-04T14:05:42.882931+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:17:49.789127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T14:05:52.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-04T14:05:52.291 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:52.291 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2023-12-04T14:05:52.303 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-04T14:05:52.142652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82580,"kb_used_data":1504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032940,"statfs":{"total":289910292480,"available":289825730560,"internally_reserved":0,"allocated":1540096,"data_stored":1278207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.246268"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:49.909082+0000","last_change":"2023-12-04T14:05:48.912820+0000","last_active":"2023-12-04T14:05:49.909082+0000","last_peered":"2023-12-04T14:05:49.909082+0000","last_clean":"2023-12-04T14:05:49.909082+0000","last_became_active":"2023-12-04T14:05:48.912549+0000","last_became_peered":"2023-12-04T14:05:48.912549+0000","last_unstale":"2023-12-04T14:05:49.909082+0000","last_undegraded":"2023-12-04T14:05:49.909082+0000","last_fullsized":"2023-12-04T14:05:49.909082+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:32:03.090571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:49.909040+0000","last_change":"2023-12-04T14:05:48.912399+0000","last_active":"2023-12-04T14:05:49.909040+0000","last_peered":"2023-12-04T14:05:49.909040+0000","last_clean":"2023-12-04T14:05:49.909040+0000","last_became_active":"2023-12-04T14:05:48.912158+0000","last_became_peered":"2023-12-04T14:05:48.912158+0000","last_unstale":"2023-12-04T14:05:49.909040+0000","last_undegraded":"2023-12-04T14:05:49.909040+0000","last_fullsized":"2023-12-04T14:05:49.909040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T23:21:47.742096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:49.908976+0000","last_change":"2023-12-04T14:05:48.911794+0000","last_active":"2023-12-04T14:05:49.908976+0000","last_peered":"2023-12-04T14:05:49.908976+0000","last_clean":"2023-12-04T14:05:49.908976+0000","last_became_active":"2023-12-04T14:05:48.911550+0000","last_became_peered":"2023-12-04T14:05:48.911550+0000","last_unstale":"2023-12-04T14:05:49.908976+0000","last_undegraded":"2023-12-04T14:05:49.908976+0000","last_fullsized":"2023-12-04T14:05:49.908976+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:22:50.543992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:49.909088+0000","last_change":"2023-12-04T14:05:48.912242+0000","last_active":"2023-12-04T14:05:49.909088+0000","last_peered":"2023-12-04T14:05:49.909088+0000","last_clean":"2023-12-04T14:05:49.909088+0000","last_became_active":"2023-12-04T14:05:48.911963+0000","last_became_peered":"2023-12-04T14:05:48.911963+0000","last_unstale":"2023-12-04T14:05:49.909088+0000","last_undegraded":"2023-12-04T14:05:49.909088+0000","last_fullsized":"2023-12-04T14:05:49.909088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T23:00:55.264982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:51.086138+0000","last_change":"2023-12-04T14:05:48.910743+0000","last_active":"2023-12-04T14:05:51.086138+0000","last_peered":"2023-12-04T14:05:51.086138+0000","last_clean":"2023-12-04T14:05:51.086138+0000","last_became_active":"2023-12-04T14:05:48.910461+0000","last_became_peered":"2023-12-04T14:05:48.910461+0000","last_unstale":"2023-12-04T14:05:51.086138+0000","last_undegraded":"2023-12-04T14:05:51.086138+0000","last_fullsized":"2023-12-04T14:05:51.086138+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T01:44:03.247744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:51.086160+0000","last_change":"2023-12-04T14:05:48.911076+0000","last_active":"2023-12-04T14:05:51.086160+0000","last_peered":"2023-12-04T14:05:51.086160+0000","last_clean":"2023-12-04T14:05:51.086160+0000","last_became_active":"2023-12-04T14:05:48.910755+0000","last_became_peered":"2023-12-04T14:05:48.910755+0000","last_unstale":"2023-12-04T14:05:51.086160+0000","last_undegraded":"2023-12-04T14:05:51.086160+0000","last_fullsized":"2023-12-04T14:05:51.086160+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:53:55.255289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:51.086138+0000","last_change":"2023-12-04T14:05:48.911361+0000","last_active":"2023-12-04T14:05:51.086138+0000","last_peered":"2023-12-04T14:05:51.086138+0000","last_clean":"2023-12-04T14:05:51.086138+0000","last_became_active":"2023-12-04T14:05:48.910748+0000","last_became_peered":"2023-12-04T14:05:48.910748+0000","last_unstale":"2023-12-04T14:05:51.086138+0000","last_undegraded":"2023-12-04T14:05:51.086138+0000","last_fullsized":"2023-12-04T14:05:51.086138+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:15:05.517055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:05:49.908936+0000","last_change":"2023-12-04T14:05:48.911395+0000","last_active":"2023-12-04T14:05:49.908936+0000","last_peered":"2023-12-04T14:05:49.908936+0000","last_clean":"2023-12-04T14:05:49.908936+0000","last_became_active":"2023-12-04T14:05:48.911100+0000","last_became_peered":"2023-12-04T14:05:48.911100+0000","last_unstale":"2023-12-04T14:05:49.908936+0000","last_undegraded":"2023-12-04T14:05:49.908936+0000","last_fullsized":"2023-12-04T14:05:49.908936+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:47.894581+0000","last_clean_scrub_stamp":"2023-12-04T14:05:47.894581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T00:43:32.547721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-04T14:05:48.905100+0000","last_change":"2023-12-04T14:05:43.891085+0000","last_active":"2023-12-04T14:05:48.905100+0000","last_peered":"2023-12-04T14:05:48.905100+0000","last_clean":"2023-12-04T14:05:48.905100+0000","last_became_active":"2023-12-04T14:05:43.890739+0000","last_became_peered":"2023-12-04T14:05:43.890739+0000","last_unstale":"2023-12-04T14:05:48.905100+0000","last_undegraded":"2023-12-04T14:05:48.905100+0000","last_fullsized":"2023-12-04T14:05:48.905100+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:05:42.882931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:05:42.882931+0000","last_clean_scrub_stamp":"2023-12-04T14:05:42.882931+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:17:49.789127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T14:05:52.304 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-04T14:05:52.304 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-12-04T14:05:52.304 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-12-04T14:05:52.304 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-12-04T14:05:52.617 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-04T14:05:52.630 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-12-04T14:05:52.630 INFO:teuthology.run_tasks:Running task ceph... 2023-12-04T14:05:52.639 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-12-04T14:05:52.639 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-04T14:05:52.639 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148', 'branch': 'wip-dis-testing-revert', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7477148', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing-revert'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-revert', 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2667, 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422', 'sleep_before_teardown': 0, 'subset': '17/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-revert', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422', 'targets': {'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCfbmNcyqeHhjNg3SpAV70mufKp07o/mtrEKQB1dLC4Q+M8Bfmnpw6lvzxhJ5bkx+i89G8/Yven4H30c2gnqfP8=', 'smithi176.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODo4dFnAcHFCBEVYKDs1fTrIEVFsW1LBi7z472ULX0gW0/1v17likGLMVd2GFpVeSPR9ljaeM34RM4QVXP6ObY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7c730b2996d0b00d56aea5ca791f90c534a13422'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-04_13:47:01', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-04T14:05:52.639 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-04T14:05:52.642 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-04T14:05:52.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2023-12-04T14:05:52.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2023-12-04T14:05:52.731 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:52.731 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2023-12-04T14:05:52.789 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-04T14:05:52.789 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2023-12-04T14:05:52.845 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-04T14:05:52.845 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T14:05:52.845 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-12-04T14:05:52.845 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T14:05:52.845 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2023-12-04T14:05:52.845 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2023-12-04 14:05:10.678700947 +0000 2023-12-04T14:05:52.845 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2023-12-04 14:02:24.502116990 +0000 2023-12-04T14:05:52.845 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2023-12-04 14:02:24.502116990 +0000 2023-12-04T14:05:52.846 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2023-12-04 14:02:24.502116990 +0000 2023-12-04T14:05:52.846 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-04T14:05:52.914 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2023-12-04T14:05:52.914 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2023-12-04T14:05:52.914 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000223855 s, 2.3 MB/s 2023-12-04T14:05:52.915 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-04T14:05:52.972 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2023-12-04T14:05:53.028 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-04T14:05:53.028 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T14:05:53.028 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-12-04T14:05:53.029 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T14:05:53.029 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2023-12-04T14:05:53.029 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2023-12-04 14:05:10.678700947 +0000 2023-12-04T14:05:53.029 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2023-12-04 14:02:24.987112955 +0000 2023-12-04T14:05:53.029 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2023-12-04 14:02:24.987112955 +0000 2023-12-04T14:05:53.029 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2023-12-04 14:02:24.987112955 +0000 2023-12-04T14:05:53.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-04T14:05:53.096 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2023-12-04T14:05:53.096 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2023-12-04T14:05:53.096 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000185976 s, 2.8 MB/s 2023-12-04T14:05:53.097 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-04T14:05:53.153 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2023-12-04T14:05:53.208 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-04T14:05:53.209 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T14:05:53.209 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 702 Links: 1 2023-12-04T14:05:53.209 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T14:05:53.209 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2023-12-04T14:05:53.209 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2023-12-04 14:05:10.679700938 +0000 2023-12-04T14:05:53.209 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2023-12-04 14:02:25.475108895 +0000 2023-12-04T14:05:53.209 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2023-12-04 14:02:25.475108895 +0000 2023-12-04T14:05:53.209 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2023-12-04 14:02:25.475108895 +0000 2023-12-04T14:05:53.209 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-04T14:05:53.274 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2023-12-04T14:05:53.275 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2023-12-04T14:05:53.275 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000171102 s, 3.0 MB/s 2023-12-04T14:05:53.276 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-04T14:05:53.331 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2023-12-04T14:05:53.386 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-04T14:05:53.387 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T14:05:53.387 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-12-04T14:05:53.387 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T14:05:53.387 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2023-12-04T14:05:53.387 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2023-12-04 14:05:10.679700938 +0000 2023-12-04T14:05:53.387 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2023-12-04 14:02:25.948104959 +0000 2023-12-04T14:05:53.387 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2023-12-04 14:02:25.948104959 +0000 2023-12-04T14:05:53.387 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2023-12-04 14:02:25.948104959 +0000 2023-12-04T14:05:53.387 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-04T14:05:53.452 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2023-12-04T14:05:53.452 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2023-12-04T14:05:53.452 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000222258 s, 2.3 MB/s 2023-12-04T14:05:53.454 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-04T14:05:53.509 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-04T14:05:53.509 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-04T14:05:53.509 INFO:tasks.ceph:Generating config... 2023-12-04T14:05:53.510 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-04T14:05:53.510 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-04T14:05:53.510 INFO:tasks.ceph:[client] log to stderr = False 2023-12-04T14:05:53.510 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-04T14:05:53.510 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2023-12-04T14:05:53.510 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:53.510 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:53.510 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:53.510 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:53.511 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:53.511 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:53.511 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:53.511 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:53.511 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-04T14:05:53.511 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-04T14:05:53.511 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-04T14:05:53.511 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-04T14:05:53.511 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-04T14:05:53.511 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-04T14:05:53.511 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-04T14:05:53.511 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-04T14:05:53.511 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-04T14:05:53.511 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-04T14:05:53.511 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-04T14:05:53.512 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-04T14:05:53.512 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-04T14:05:53.512 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-04T14:05:53.512 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-04T14:05:53.512 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-04T14:05:53.512 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-04T14:05:53.512 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-04T14:05:53.512 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-04T14:05:53.512 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-04T14:05:53.512 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-04T14:05:53.512 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-04T14:05:53.512 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-04T14:05:53.512 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-12-04T14:05:53.513 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-12-04T14:05:53.591 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster2.keyring 2023-12-04T14:05:53.593 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-12-04T14:05:53.676 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-04T14:05:53.743 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.176')] 2023-12-04T14:05:53.743 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.176', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-04T14:05:53.744 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:53.744 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-04T14:05:53.798 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.176 --print /home/ubuntu/cephtest/cluster2.monmap 2023-12-04T14:05:53.877 INFO:teuthology.orchestra.run.smithi176.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-12-04T14:05:53.877 INFO:teuthology.orchestra.run.smithi176.stdout:monmaptool: generated fsid 3dd73e37-8026-4c88-8202-aea0e5eb7612 2023-12-04T14:05:53.877 INFO:teuthology.orchestra.run.smithi176.stdout:setting min_mon_release = pacific 2023-12-04T14:05:53.877 INFO:teuthology.orchestra.run.smithi176.stdout:epoch 0 2023-12-04T14:05:53.877 INFO:teuthology.orchestra.run.smithi176.stdout:fsid 3dd73e37-8026-4c88-8202-aea0e5eb7612 2023-12-04T14:05:53.877 INFO:teuthology.orchestra.run.smithi176.stdout:last_changed 2023-12-04T14:05:53.876430+0000 2023-12-04T14:05:53.877 INFO:teuthology.orchestra.run.smithi176.stdout:created 2023-12-04T14:05:53.876430+0000 2023-12-04T14:05:53.877 INFO:teuthology.orchestra.run.smithi176.stdout:min_mon_release 16 (pacific) 2023-12-04T14:05:53.877 INFO:teuthology.orchestra.run.smithi176.stdout:election_strategy: 1 2023-12-04T14:05:53.877 INFO:teuthology.orchestra.run.smithi176.stdout:0: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.a 2023-12-04T14:05:53.877 INFO:teuthology.orchestra.run.smithi176.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-12-04T14:05:53.878 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-04T14:05:53.933 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 3dd73e37-8026-4c88-8202-aea0e5eb7612... 2023-12-04T14:05:53.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-04T14:05:53.936 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-04T14:05:53.991 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2023-12-04T14:05:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2023-12-04T14:05:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-04T14:05:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-04T14:05:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = 172.21.15.176 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: mon client directed command retry = 5 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on pool no app = False 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: ms inject socket failures = 5000 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 3dd73e37-8026-4c88-8202-aea0e5eb7612 2023-12-04T14:05:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2023-12-04T14:05:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = stupid 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2023-12-04T14:05:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2023-12-04T14:05:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2023-12-04T14:05:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: debug rbd = 20 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: debug rbd_mirror = 15 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: log to stderr = False 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: rbd default features = 125 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: rbd mirror image policy type = none 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.0] 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:54.001 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.1] 2023-12-04T14:05:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:54.001 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.2] 2023-12-04T14:05:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:54.001 INFO:teuthology.orchestra.run.smithi146.stdout:[client.mirror.3] 2023-12-04T14:05:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:54.022 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2023-12-04T14:05:54.022 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2023-12-04T14:05:54.022 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-04T14:05:54.022 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2023-12-04T14:05:54.022 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.022 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-04T14:05:54.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: mon_allow_pool_size_one = true 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-04T14:05:54.024 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = 172.21.15.176 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: mon client directed command retry = 5 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on pool no app = False 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject socket failures = 5000 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = 3dd73e37-8026-4c88-8202-aea0e5eb7612 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.025 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock profile = high_recovery_ops 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug trim objects = true 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2023-12-04T14:05:54.026 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: bdev async discard = True 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: bdev enable discard = True 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore allocator = stupid 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2023-12-04T14:05:54.027 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 1/20 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 1/20 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 4/10 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2023-12-04T14:05:54.028 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2023-12-04T14:05:54.029 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: # 1m isn't quite enough 2023-12-04T14:05:54.030 INFO:teuthology.orchestra.run.smithi176.stdout: mon_down_mkfs_grace = 2m 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_filestore_osds = false 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: debug rbd = 20 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: debug rbd_mirror = 15 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: log to stderr = False 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: rbd default features = 125 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: rbd mirror image policy type = none 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.0] 2023-12-04T14:05:54.031 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:54.032 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:54.032 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.1] 2023-12-04T14:05:54.032 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:54.032 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:54.032 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.2] 2023-12-04T14:05:54.032 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:54.032 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:54.032 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.3] 2023-12-04T14:05:54.032 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-04T14:05:54.032 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-04T14:05:54.036 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-12-04T14:05:54.036 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-12-04T14:05:54.131 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-04T14:05:54.131 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:54.131 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-12-04T14:05:54.145 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:54.145 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-12-04T14:05:54.200 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2023-12-04T14:05:54.200 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:54.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-04T14:05:54.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-04T14:05:54.244 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:05:54.244 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-04T14:05:54.300 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2023-12-04T14:05:54.301 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:54.301 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-04T14:05:54.301 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-04T14:05:54.339 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:05:54.339 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-04T14:05:54.394 INFO:tasks.ceph:Setting up mon nodes... 2023-12-04T14:05:54.394 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-04T14:05:54.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-04T14:05:54.452 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-04T14:05:54.455 INFO:tasks.ceph:Setting up mds nodes... 2023-12-04T14:05:54.455 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-04T14:05:54.455 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-12-04T14:05:54.498 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-12-04T14:05:54.513 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-12-04T14:05:54.555 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-12-04T14:05:54.570 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-04T14:05:54.651 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-04T14:05:54.666 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-04T14:05:54.749 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-04T14:05:54.764 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-04T14:05:54.846 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-04T14:05:54.861 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-04T14:05:54.942 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-04T14:05:54.957 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-04T14:05:55.039 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-04T14:05:55.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-04T14:05:55.136 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-04T14:05:55.151 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-04T14:05:55.233 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-04T14:05:55.248 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-04T14:05:55.248 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-04T14:05:55.249 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-12-04T14:05:55.315 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-04T14:05:55.315 INFO:tasks.ceph:role: cluster2.osd.0 2023-12-04T14:05:55.315 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com 2023-12-04T14:05:55.315 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-04T14:05:55.389 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T14:05:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T14:05:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T14:05:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-04T14:05:55.389 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T14:05:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2023-12-04T14:05:55.389 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T14:05:55.389 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-04T14:05:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T14:05:55.390 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T14:05:55.905 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2023-12-04T14:05:55.907 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2023-12-04T14:05:55.907 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-12-04T14:05:56.047 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-12-04T14:05:56.081 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-12-04T14:05:56.147 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-04T14:05:56.147 INFO:tasks.ceph:role: cluster2.osd.1 2023-12-04T14:05:56.147 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com 2023-12-04T14:05:56.147 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-04T14:05:56.221 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T14:05:56.221 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T14:05:56.221 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T14:05:56.221 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-04T14:05:56.221 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T14:05:56.221 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2023-12-04T14:05:56.222 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T14:05:56.222 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-04T14:05:56.222 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T14:05:56.222 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T14:05:56.682 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2023-12-04T14:05:56.684 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2023-12-04T14:05:56.685 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-12-04T14:05:56.767 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-12-04T14:05:56.839 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-12-04T14:05:56.906 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-04T14:05:56.906 INFO:tasks.ceph:role: cluster2.osd.2 2023-12-04T14:05:56.906 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com 2023-12-04T14:05:56.907 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-04T14:05:56.980 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T14:05:56.980 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T14:05:56.980 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T14:05:56.981 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-04T14:05:56.981 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T14:05:56.981 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2023-12-04T14:05:56.981 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T14:05:56.981 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-04T14:05:56.981 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T14:05:56.981 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T14:05:57.474 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2023-12-04T14:05:57.476 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2023-12-04T14:05:57.476 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-12-04T14:05:57.614 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-12-04T14:05:57.645 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-04T14:05:57.744 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:05:57.742+0000 7f8eaef82500 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-12-04T14:05:57.746 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:05:57.744+0000 7f8eaef82500 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-12-04T14:05:57.746 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:05:57.744+0000 7f8eaef82500 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-04T14:05:57.746 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:05:57.744+0000 7f8eaef82500 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-04T14:05:57.746 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:05:57.744+0000 7f8eaef82500 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-04T14:05:57.753 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:05:57.752+0000 7f8eaef82500 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-12-04T14:06:00.600 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-12-04T14:06:00.632 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-04T14:06:00.724 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:06:00.721+0000 7f8d2360e500 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-12-04T14:06:00.724 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:06:00.722+0000 7f8d2360e500 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-12-04T14:06:00.724 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:06:00.722+0000 7f8d2360e500 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-04T14:06:00.724 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:06:00.722+0000 7f8d2360e500 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-04T14:06:00.724 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:06:00.722+0000 7f8d2360e500 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-04T14:06:00.724 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:06:00.722+0000 7f8d2360e500 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-12-04T14:06:03.576 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-12-04T14:06:03.608 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-04T14:06:03.701 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:06:03.700+0000 7f5ba660d500 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-12-04T14:06:03.701 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:06:03.700+0000 7f5ba660d500 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-12-04T14:06:03.701 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:06:03.700+0000 7f5ba660d500 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-04T14:06:03.702 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:06:03.700+0000 7f5ba660d500 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-04T14:06:03.702 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:06:03.700+0000 7f5ba660d500 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-04T14:06:03.702 INFO:teuthology.orchestra.run.smithi176.stderr:2023-12-04T14:06:03.701+0000 7f5ba660d500 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-12-04T14:06:06.547 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-12-04T14:06:06.578 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-04T14:06:06.579 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:06:06.579 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-12-04T14:06:06.611 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:06.611 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-12-04T14:06:06.649 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:06.649 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-12-04T14:06:06.719 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:06.719 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-12-04T14:06:06.787 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-12-04T14:06:06.787 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-12-04T14:06:06.805 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:06.806 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-12-04T14:06:06.844 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:06.844 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-12-04T14:06:06.900 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:06.900 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-12-04T14:06:06.956 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:06.956 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-12-04T14:06:07.010 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:07.010 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-12-04T14:06:07.065 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:07.066 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-12-04T14:06:07.120 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:07.120 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-12-04T14:06:07.175 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:07.175 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-12-04T14:06:07.230 INFO:tasks.ceph:Adding keys to all mons... 2023-12-04T14:06:07.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo tee -a /etc/ceph/cluster2.keyring 2023-12-04T14:06:07.296 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.x] 2023-12-04T14:06:07.296 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDC3G1laIDxGhAAG/ZWMCN96bU+Z6OXX2ItZg== 2023-12-04T14:06:07.296 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.0] 2023-12-04T14:06:07.296 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDF3G1l1vBWLBAAv0HX40IQG2+1mkCqZe5dcg== 2023-12-04T14:06:07.296 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.1] 2023-12-04T14:06:07.296 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDI3G1lYHIaKxAALtkALQAK7pbfV/535J4OsA== 2023-12-04T14:06:07.296 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.2] 2023-12-04T14:06:07.296 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDL3G1lQhvGKRAAsn6nxOMQEqj+qTgWzAxPVA== 2023-12-04T14:06:07.297 INFO:teuthology.orchestra.run.smithi176.stdout:[client.0] 2023-12-04T14:06:07.297 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDC3G1lt9muHRAAYliEBM+tfCZm3/aOp+G/aQ== 2023-12-04T14:06:07.297 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror] 2023-12-04T14:06:07.297 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDC3G1lhBcTIRAA5Hf5VUuMOUloYfJliE0wVg== 2023-12-04T14:06:07.297 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.0] 2023-12-04T14:06:07.297 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDC3G1lm07SJhAAE0L2yQrvx0IVtCouka2f0g== 2023-12-04T14:06:07.297 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.1] 2023-12-04T14:06:07.297 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDC3G1lK9OkLBAA21OQ/ogYXHaFPWE8gVUT9g== 2023-12-04T14:06:07.297 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.2] 2023-12-04T14:06:07.297 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDC3G1lw31pMhAA9zzJiWK0A4y3viPOBfhD9g== 2023-12-04T14:06:07.297 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.3] 2023-12-04T14:06:07.298 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDC3G1liwEnOBAAPFhr39VCC6ujBmu05ZfwmQ== 2023-12-04T14:06:07.298 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.4] 2023-12-04T14:06:07.298 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDD3G1lJkdUAhAAqjrjqGgJwIDLNmCmybYxmQ== 2023-12-04T14:06:07.298 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.5] 2023-12-04T14:06:07.298 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDD3G1l0KgcCBAAyvS/K9nHOc9R2Yr1PMJBnQ== 2023-12-04T14:06:07.298 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.6] 2023-12-04T14:06:07.298 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDD3G1lCn/pDRAAnOY6mjxmsRbd6awAzTjCsA== 2023-12-04T14:06:07.299 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-04T14:06:07.391 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T14:06:07.446 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T14:06:07.498 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T14:06:07.550 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:06:07.604 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:06:07.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:06:07.714 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:06:07.769 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:06:07.824 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:06:07.878 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:06:07.933 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:06:07.988 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T14:06:08.042 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-04T14:06:08.043 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-12-04T14:06:08.070 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-12-04T14:06:08.166 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-12-04T14:06:08.192 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-12-04T14:06:08.247 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-12-04T14:06:08.247 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-12-04T14:06:08.247 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-12-04T14:06:08.289 INFO:tasks.ceph.cluster2.mon.a:Started 2023-12-04T14:06:08.289 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-12-04T14:06:08.289 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-12-04T14:06:08.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-12-04T14:06:08.292 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-12-04T14:06:08.292 INFO:tasks.ceph:Setting crush tunables to default 2023-12-04T14:06:08.293 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster cluster2 osd crush tunables default 2023-12-04T14:06:08.482 INFO:teuthology.orchestra.run.smithi176.stderr:adjusted tunables profile to default 2023-12-04T14:06:08.493 INFO:tasks.ceph:check_enable_crimson: False 2023-12-04T14:06:08.493 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-12-04T14:06:08.493 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:08.493 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-12-04T14:06:08.519 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:08.519 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-12-04T14:06:08.586 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:06:08.586 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-12-04T14:06:08.652 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster cluster2 osd new 8e7ca286-19e8-43d0-8ab0-0b58d2b00b90 0 2023-12-04T14:06:08.745 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:08.744+0000 7f15da411040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-04T14:06:08.790 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:08.789+0000 7f15da411040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-04T14:06:08.867 INFO:teuthology.orchestra.run.smithi176.stdout:0 2023-12-04T14:06:08.878 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster cluster2 osd new fa1dc8fd-dbc1-4d4e-b402-83ecbcc5e26d 1 2023-12-04T14:06:09.053 INFO:teuthology.orchestra.run.smithi176.stdout:1 2023-12-04T14:06:09.064 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster cluster2 osd new 5647fecf-82f9-45f0-919a-d721e683f8f8 2 2023-12-04T14:06:09.182 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:09.180+0000 7f15da411040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-04T14:06:09.225 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:09.224+0000 7f15da411040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-04T14:06:09.239 INFO:teuthology.orchestra.run.smithi176.stdout:2 2023-12-04T14:06:09.250 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-12-04T14:06:09.251 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-12-04T14:06:09.253 INFO:tasks.ceph.cluster2.osd.0:Started 2023-12-04T14:06:09.253 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-12-04T14:06:09.253 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-12-04T14:06:09.255 INFO:tasks.ceph.cluster2.osd.1:Started 2023-12-04T14:06:09.255 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-12-04T14:06:09.255 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-12-04T14:06:09.257 INFO:tasks.ceph.cluster2.osd.2:Started 2023-12-04T14:06:09.257 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-04T14:06:09.268 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:09.266+0000 7f15da411040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-04T14:06:09.319 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:09.317+0000 7f15da411040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-04T14:06:09.361 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:09.359+0000 7f15da411040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-04T14:06:09.428 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:06:09.428 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":5,"fsid":"3dd73e37-8026-4c88-8202-aea0e5eb7612","created":"2023-12-04T14:06:08.402075+0000","modified":"2023-12-04T14:06:09.238687+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T14:06:09.238687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8e7ca286-19e8-43d0-8ab0-0b58d2b00b90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fa1dc8fd-dbc1-4d4e-b402-83ecbcc5e26d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5647fecf-82f9-45f0-919a-d721e683f8f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T14:06:09.439 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-12-04T14:06:09.439 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-04T14:06:09.459 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:09.458+0000 7f15da411040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-04T14:06:09.512 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:09.511+0000 7f15da411040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-04T14:06:09.554 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:09.552+0000 7f15da411040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-04T14:06:09.602 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:09.601+0000 7f15da411040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-04T14:06:09.644 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:09.643+0000 7f15da411040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-04T14:06:09.742 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-04T14:06:09.853 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:06:09.852+0000 7f8e5c4c9500 -1 Falling back to public interface 2023-12-04T14:06:09.854 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:06:09.853+0000 7f0539c0d500 -1 Falling back to public interface 2023-12-04T14:06:09.862 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:06:09.861+0000 7f4c0c41d500 -1 Falling back to public interface 2023-12-04T14:06:09.901 INFO:teuthology.misc.health.smithi176.stdout: 2023-12-04T14:06:09.902 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":5,"fsid":"3dd73e37-8026-4c88-8202-aea0e5eb7612","created":"2023-12-04T14:06:08.402075+0000","modified":"2023-12-04T14:06:09.238687+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T14:06:09.238687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8e7ca286-19e8-43d0-8ab0-0b58d2b00b90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fa1dc8fd-dbc1-4d4e-b402-83ecbcc5e26d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5647fecf-82f9-45f0-919a-d721e683f8f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T14:06:09.912 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-04T14:06:09.960 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:09.959+0000 7f15da411040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-04T14:06:10.090 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:10.088+0000 7f15da411040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-04T14:06:10.181 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:10.179+0000 7f15da411040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-04T14:06:10.223 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:10.221+0000 7f15da411040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-04T14:06:10.317 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:10.315+0000 7f15da411040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-04T14:06:10.407 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:10.405+0000 7f15da411040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-04T14:06:10.543 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:10.541+0000 7f15da411040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-04T14:06:10.584 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:10.583+0000 7f15da411040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-04T14:06:10.687 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:10.686+0000 7f15da411040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-04T14:06:11.284 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:11.282+0000 7f15da411040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-04T14:06:11.547 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:11.545+0000 7f15da411040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-04T14:06:11.595 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:11.593+0000 7f15da411040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-04T14:06:11.694 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:11.692+0000 7f15da411040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-04T14:06:12.714 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:06:12.712+0000 7f157bf45640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-04T14:06:13.315 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:06:13.314+0000 7f8e5c4c9500 -1 osd.0 0 log_to_monitors true 2023-12-04T14:06:13.341 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:06:13.340+0000 7f0539c0d500 -1 osd.1 0 log_to_monitors true 2023-12-04T14:06:13.359 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:06:13.359+0000 7f4c0c41d500 -1 osd.2 0 log_to_monitors true 2023-12-04T14:06:16.215 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-04T14:06:16.495 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:06:16.494+0000 7f4c08d76640 -1 osd.2 0 waiting for initial osdmap 2023-12-04T14:06:16.508 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:06:16.507+0000 7f4bffb6f640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T14:06:16.540 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:06:16.539+0000 7f05365f1640 -1 osd.1 0 waiting for initial osdmap 2023-12-04T14:06:16.557 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:06:16.556+0000 7f052d3ea640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T14:06:16.596 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:06:16.595+0000 7f8e57c32640 -1 osd.0 0 waiting for initial osdmap 2023-12-04T14:06:16.602 INFO:teuthology.misc.health.smithi176.stdout: 2023-12-04T14:06:16.602 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":7,"fsid":"3dd73e37-8026-4c88-8202-aea0e5eb7612","created":"2023-12-04T14:06:08.402075+0000","modified":"2023-12-04T14:06:14.711670+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T14:06:09.238687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8e7ca286-19e8-43d0-8ab0-0b58d2b00b90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fa1dc8fd-dbc1-4d4e-b402-83ecbcc5e26d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5647fecf-82f9-45f0-919a-d721e683f8f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T14:06:16.613 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-04T14:06:16.614 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:06:16.613+0000 7f8e50265640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T14:06:22.917 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-04T14:06:23.195 INFO:teuthology.misc.health.smithi176.stdout: 2023-12-04T14:06:23.195 INFO:teuthology.misc.health.smithi176.stdout:{"epoch":11,"fsid":"3dd73e37-8026-4c88-8202-aea0e5eb7612","created":"2023-12-04T14:06:08.402075+0000","modified":"2023-12-04T14:06:20.735861+0000","last_up_change":"2023-12-04T14:06:17.498646+0000","last_in_change":"2023-12-04T14:06:09.238687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T14:06:17.754836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8e7ca286-19e8-43d0-8ab0-0b58d2b00b90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":2137896407},{"type":"v1","addr":"172.21.15.176:6801","nonce":2137896407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":2137896407},{"type":"v1","addr":"172.21.15.176:6803","nonce":2137896407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":2137896407},{"type":"v1","addr":"172.21.15.176:6807","nonce":2137896407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":2137896407},{"type":"v1","addr":"172.21.15.176:6805","nonce":2137896407}]},"public_addr":"172.21.15.176:6801/2137896407","cluster_addr":"172.21.15.176:6803/2137896407","heartbeat_back_addr":"172.21.15.176:6807/2137896407","heartbeat_front_addr":"172.21.15.176:6805/2137896407","state":["exists","up"]},{"osd":1,"uuid":"fa1dc8fd-dbc1-4d4e-b402-83ecbcc5e26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2779676088},{"type":"v1","addr":"172.21.15.176:6809","nonce":2779676088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2779676088},{"type":"v1","addr":"172.21.15.176:6811","nonce":2779676088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2779676088},{"type":"v1","addr":"172.21.15.176:6815","nonce":2779676088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2779676088},{"type":"v1","addr":"172.21.15.176:6813","nonce":2779676088}]},"public_addr":"172.21.15.176:6809/2779676088","cluster_addr":"172.21.15.176:6811/2779676088","heartbeat_back_addr":"172.21.15.176:6815/2779676088","heartbeat_front_addr":"172.21.15.176:6813/2779676088","state":["exists","up"]},{"osd":2,"uuid":"5647fecf-82f9-45f0-919a-d721e683f8f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":1210687444},{"type":"v1","addr":"172.21.15.176:6817","nonce":1210687444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":1210687444},{"type":"v1","addr":"172.21.15.176:6819","nonce":1210687444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":1210687444},{"type":"v1","addr":"172.21.15.176:6823","nonce":1210687444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":1210687444},{"type":"v1","addr":"172.21.15.176:6821","nonce":1210687444}]},"public_addr":"172.21.15.176:6817/1210687444","cluster_addr":"172.21.15.176:6819/1210687444","heartbeat_back_addr":"172.21.15.176:6823/1210687444","heartbeat_front_addr":"172.21.15.176:6821/1210687444","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:06:14.308601+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:06:14.358601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:06:14.341890+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T14:06:23.207 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-04T14:06:23.207 INFO:tasks.ceph:Creating RBD pool 2023-12-04T14:06:23.207 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-12-04T14:06:24.377 INFO:teuthology.orchestra.run.smithi176.stderr:pool 'rbd' created 2023-12-04T14:06:24.393 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-04T14:06:25.381 INFO:teuthology.orchestra.run.smithi176.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-04T14:06:25.400 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-12-04T14:06:25.401 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-12-04T14:06:25.401 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-12-04T14:06:25.401 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-12-04T14:06:25.401 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-04T14:06:25.726 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:06:25.738 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":1922868602},{"type":"v1","addr":"172.21.15.146:6827","nonce":1922868602}]},"active_addr":"172.21.15.146:6827/1922868602","active_change":"2023-12-04T14:06:11.707012+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":3842243363}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":3444989832}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":3130679109}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":164503865}]}]} 2023-12-04T14:06:25.740 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-12-04T14:06:25.740 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-12-04T14:06:25.740 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-04T14:06:26.017 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:06:26.018 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":13,"fsid":"3dd73e37-8026-4c88-8202-aea0e5eb7612","created":"2023-12-04T14:06:08.402075+0000","modified":"2023-12-04T14:06:25.376951+0000","last_up_change":"2023-12-04T14:06:17.498646+0000","last_in_change":"2023-12-04T14:06:09.238687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T14:06:17.754836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-04T14:06:23.501553+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8e7ca286-19e8-43d0-8ab0-0b58d2b00b90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":2137896407},{"type":"v1","addr":"172.21.15.176:6801","nonce":2137896407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":2137896407},{"type":"v1","addr":"172.21.15.176:6803","nonce":2137896407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":2137896407},{"type":"v1","addr":"172.21.15.176:6807","nonce":2137896407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":2137896407},{"type":"v1","addr":"172.21.15.176:6805","nonce":2137896407}]},"public_addr":"172.21.15.176:6801/2137896407","cluster_addr":"172.21.15.176:6803/2137896407","heartbeat_back_addr":"172.21.15.176:6807/2137896407","heartbeat_front_addr":"172.21.15.176:6805/2137896407","state":["exists","up"]},{"osd":1,"uuid":"fa1dc8fd-dbc1-4d4e-b402-83ecbcc5e26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2779676088},{"type":"v1","addr":"172.21.15.176:6809","nonce":2779676088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2779676088},{"type":"v1","addr":"172.21.15.176:6811","nonce":2779676088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2779676088},{"type":"v1","addr":"172.21.15.176:6815","nonce":2779676088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2779676088},{"type":"v1","addr":"172.21.15.176:6813","nonce":2779676088}]},"public_addr":"172.21.15.176:6809/2779676088","cluster_addr":"172.21.15.176:6811/2779676088","heartbeat_back_addr":"172.21.15.176:6815/2779676088","heartbeat_front_addr":"172.21.15.176:6813/2779676088","state":["exists","up"]},{"osd":2,"uuid":"5647fecf-82f9-45f0-919a-d721e683f8f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":1210687444},{"type":"v1","addr":"172.21.15.176:6817","nonce":1210687444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":1210687444},{"type":"v1","addr":"172.21.15.176:6819","nonce":1210687444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":1210687444},{"type":"v1","addr":"172.21.15.176:6823","nonce":1210687444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":1210687444},{"type":"v1","addr":"172.21.15.176:6821","nonce":1210687444}]},"public_addr":"172.21.15.176:6817/1210687444","cluster_addr":"172.21.15.176:6819/1210687444","heartbeat_back_addr":"172.21.15.176:6823/1210687444","heartbeat_front_addr":"172.21.15.176:6821/1210687444","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:06:14.308601+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:06:14.358601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:06:14.341890+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T14:06:26.030 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-12-04T14:06:26.030 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-04T14:06:26.306 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:06:26.306 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":13,"fsid":"3dd73e37-8026-4c88-8202-aea0e5eb7612","created":"2023-12-04T14:06:08.402075+0000","modified":"2023-12-04T14:06:25.376951+0000","last_up_change":"2023-12-04T14:06:17.498646+0000","last_in_change":"2023-12-04T14:06:09.238687+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T14:06:17.754836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-04T14:06:23.501553+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8e7ca286-19e8-43d0-8ab0-0b58d2b00b90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":2137896407},{"type":"v1","addr":"172.21.15.176:6801","nonce":2137896407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":2137896407},{"type":"v1","addr":"172.21.15.176:6803","nonce":2137896407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":2137896407},{"type":"v1","addr":"172.21.15.176:6807","nonce":2137896407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":2137896407},{"type":"v1","addr":"172.21.15.176:6805","nonce":2137896407}]},"public_addr":"172.21.15.176:6801/2137896407","cluster_addr":"172.21.15.176:6803/2137896407","heartbeat_back_addr":"172.21.15.176:6807/2137896407","heartbeat_front_addr":"172.21.15.176:6805/2137896407","state":["exists","up"]},{"osd":1,"uuid":"fa1dc8fd-dbc1-4d4e-b402-83ecbcc5e26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2779676088},{"type":"v1","addr":"172.21.15.176:6809","nonce":2779676088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2779676088},{"type":"v1","addr":"172.21.15.176:6811","nonce":2779676088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2779676088},{"type":"v1","addr":"172.21.15.176:6815","nonce":2779676088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2779676088},{"type":"v1","addr":"172.21.15.176:6813","nonce":2779676088}]},"public_addr":"172.21.15.176:6809/2779676088","cluster_addr":"172.21.15.176:6811/2779676088","heartbeat_back_addr":"172.21.15.176:6815/2779676088","heartbeat_front_addr":"172.21.15.176:6813/2779676088","state":["exists","up"]},{"osd":2,"uuid":"5647fecf-82f9-45f0-919a-d721e683f8f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":1210687444},{"type":"v1","addr":"172.21.15.176:6817","nonce":1210687444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":1210687444},{"type":"v1","addr":"172.21.15.176:6819","nonce":1210687444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":1210687444},{"type":"v1","addr":"172.21.15.176:6823","nonce":1210687444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":1210687444},{"type":"v1","addr":"172.21.15.176:6821","nonce":1210687444}]},"public_addr":"172.21.15.176:6817/1210687444","cluster_addr":"172.21.15.176:6819/1210687444","heartbeat_back_addr":"172.21.15.176:6823/1210687444","heartbeat_front_addr":"172.21.15.176:6821/1210687444","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:06:14.308601+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:06:14.358601+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T14:06:14.341890+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T14:06:26.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-12-04T14:06:26.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-12-04T14:06:26.320 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-12-04T14:06:26.461 INFO:teuthology.orchestra.run.smithi176.stdout:34359738374 2023-12-04T14:06:26.462 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-04T14:06:26.462 INFO:teuthology.orchestra.run.smithi176.stdout:34359738374 2023-12-04T14:06:26.463 INFO:teuthology.orchestra.run.smithi176.stdout:34359738374 2023-12-04T14:06:26.463 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-04T14:06:26.463 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-04T14:06:26.798 INFO:teuthology.orchestra.run.smithi176.stdout:34359738373 2023-12-04T14:06:26.802 INFO:teuthology.orchestra.run.smithi176.stdout:34359738373 2023-12-04T14:06:26.807 INFO:teuthology.orchestra.run.smithi176.stdout:34359738373 2023-12-04T14:06:26.812 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2023-12-04T14:06:26.819 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2023-12-04T14:06:26.821 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2023-12-04T14:06:27.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-04T14:06:27.819 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-04T14:06:27.822 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-04T14:06:28.134 INFO:teuthology.orchestra.run.smithi176.stdout:34359738373 2023-12-04T14:06:28.147 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2023-12-04T14:06:28.161 INFO:teuthology.orchestra.run.smithi176.stdout:34359738373 2023-12-04T14:06:28.172 INFO:teuthology.orchestra.run.smithi176.stdout:34359738373 2023-12-04T14:06:28.175 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2023-12-04T14:06:28.185 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2023-12-04T14:06:29.148 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-04T14:06:29.176 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-04T14:06:29.187 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-04T14:06:29.455 INFO:teuthology.orchestra.run.smithi176.stdout:34359738374 2023-12-04T14:06:29.468 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.0 2023-12-04T14:06:29.468 DEBUG:teuthology.parallel:result is None 2023-12-04T14:06:29.489 INFO:teuthology.orchestra.run.smithi176.stdout:34359738374 2023-12-04T14:06:29.496 INFO:teuthology.orchestra.run.smithi176.stdout:34359738374 2023-12-04T14:06:29.502 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2023-12-04T14:06:29.503 DEBUG:teuthology.parallel:result is None 2023-12-04T14:06:29.509 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.2 2023-12-04T14:06:29.509 DEBUG:teuthology.parallel:result is None 2023-12-04T14:06:29.510 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-04T14:06:29.510 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-04T14:06:29.825 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:06:29.826 INFO:teuthology.orchestra.run.smithi176.stderr:dumped all 2023-12-04T14:06:29.837 INFO:teuthology.orchestra.run.smithi176.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-04T14:06:29.714119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82600,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032920,"statfs":{"total":289910292480,"available":289825710080,"internally_reserved":0,"allocated":1556480,"data_stored":1285163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001292"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:26.744574+0000","last_change":"2023-12-04T14:06:25.388466+0000","last_active":"2023-12-04T14:06:26.744574+0000","last_peered":"2023-12-04T14:06:26.744574+0000","last_clean":"2023-12-04T14:06:26.744574+0000","last_became_active":"2023-12-04T14:06:25.388163+0000","last_became_peered":"2023-12-04T14:06:25.388163+0000","last_unstale":"2023-12-04T14:06:26.744574+0000","last_undegraded":"2023-12-04T14:06:26.744574+0000","last_fullsized":"2023-12-04T14:06:26.744574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:17:26.277565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:26.744570+0000","last_change":"2023-12-04T14:06:25.390563+0000","last_active":"2023-12-04T14:06:26.744570+0000","last_peered":"2023-12-04T14:06:26.744570+0000","last_clean":"2023-12-04T14:06:26.744570+0000","last_became_active":"2023-12-04T14:06:25.390081+0000","last_became_peered":"2023-12-04T14:06:25.390081+0000","last_unstale":"2023-12-04T14:06:26.744570+0000","last_undegraded":"2023-12-04T14:06:26.744570+0000","last_fullsized":"2023-12-04T14:06:26.744570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:37:41.613703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:26.744697+0000","last_change":"2023-12-04T14:06:25.390097+0000","last_active":"2023-12-04T14:06:26.744697+0000","last_peered":"2023-12-04T14:06:26.744697+0000","last_clean":"2023-12-04T14:06:26.744697+0000","last_became_active":"2023-12-04T14:06:25.389613+0000","last_became_peered":"2023-12-04T14:06:25.389613+0000","last_unstale":"2023-12-04T14:06:26.744697+0000","last_undegraded":"2023-12-04T14:06:26.744697+0000","last_fullsized":"2023-12-04T14:06:26.744697+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:39:08.473833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:26.744575+0000","last_change":"2023-12-04T14:06:25.388414+0000","last_active":"2023-12-04T14:06:26.744575+0000","last_peered":"2023-12-04T14:06:26.744575+0000","last_clean":"2023-12-04T14:06:26.744575+0000","last_became_active":"2023-12-04T14:06:25.388105+0000","last_became_peered":"2023-12-04T14:06:25.388105+0000","last_unstale":"2023-12-04T14:06:26.744575+0000","last_undegraded":"2023-12-04T14:06:26.744575+0000","last_fullsized":"2023-12-04T14:06:26.744575+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:51:17.379730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:26.386193+0000","last_change":"2023-12-04T14:06:25.388568+0000","last_active":"2023-12-04T14:06:26.386193+0000","last_peered":"2023-12-04T14:06:26.386193+0000","last_clean":"2023-12-04T14:06:26.386193+0000","last_became_active":"2023-12-04T14:06:25.387970+0000","last_became_peered":"2023-12-04T14:06:25.387970+0000","last_unstale":"2023-12-04T14:06:26.386193+0000","last_undegraded":"2023-12-04T14:06:26.386193+0000","last_fullsized":"2023-12-04T14:06:26.386193+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T01:29:31.446711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:27.244861+0000","last_change":"2023-12-04T14:06:25.388574+0000","last_active":"2023-12-04T14:06:27.244861+0000","last_peered":"2023-12-04T14:06:27.244861+0000","last_clean":"2023-12-04T14:06:27.244861+0000","last_became_active":"2023-12-04T14:06:25.388254+0000","last_became_peered":"2023-12-04T14:06:25.388254+0000","last_unstale":"2023-12-04T14:06:27.244861+0000","last_undegraded":"2023-12-04T14:06:27.244861+0000","last_fullsized":"2023-12-04T14:06:27.244861+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:40:08.774747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:27.244849+0000","last_change":"2023-12-04T14:06:25.389261+0000","last_active":"2023-12-04T14:06:27.244849+0000","last_peered":"2023-12-04T14:06:27.244849+0000","last_clean":"2023-12-04T14:06:27.244849+0000","last_became_active":"2023-12-04T14:06:25.388830+0000","last_became_peered":"2023-12-04T14:06:25.388830+0000","last_unstale":"2023-12-04T14:06:27.244849+0000","last_undegraded":"2023-12-04T14:06:27.244849+0000","last_fullsized":"2023-12-04T14:06:27.244849+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:19:33.194202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:26.744651+0000","last_change":"2023-12-04T14:06:25.388372+0000","last_active":"2023-12-04T14:06:26.744651+0000","last_peered":"2023-12-04T14:06:26.744651+0000","last_clean":"2023-12-04T14:06:26.744651+0000","last_became_active":"2023-12-04T14:06:25.388074+0000","last_became_peered":"2023-12-04T14:06:25.388074+0000","last_unstale":"2023-12-04T14:06:26.744651+0000","last_undegraded":"2023-12-04T14:06:26.744651+0000","last_fullsized":"2023-12-04T14:06:26.744651+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:31:08.592828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-04T14:06:25.386130+0000","last_change":"2023-12-04T14:06:19.745903+0000","last_active":"2023-12-04T14:06:25.386130+0000","last_peered":"2023-12-04T14:06:25.386130+0000","last_clean":"2023-12-04T14:06:25.386130+0000","last_became_active":"2023-12-04T14:06:19.745538+0000","last_became_peered":"2023-12-04T14:06:19.745538+0000","last_unstale":"2023-12-04T14:06:25.386130+0000","last_undegraded":"2023-12-04T14:06:25.386130+0000","last_fullsized":"2023-12-04T14:06:25.386130+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:18.731113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:18.731113+0000","last_clean_scrub_stamp":"2023-12-04T14:06:18.731113+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T23:32:15.394300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":34809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T14:06:29.838 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-04T14:06:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:06:30.115 INFO:teuthology.orchestra.run.smithi176.stderr:dumped all 2023-12-04T14:06:30.127 INFO:teuthology.orchestra.run.smithi176.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-04T14:06:29.714119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82600,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032920,"statfs":{"total":289910292480,"available":289825710080,"internally_reserved":0,"allocated":1556480,"data_stored":1285163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001292"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:26.744574+0000","last_change":"2023-12-04T14:06:25.388466+0000","last_active":"2023-12-04T14:06:26.744574+0000","last_peered":"2023-12-04T14:06:26.744574+0000","last_clean":"2023-12-04T14:06:26.744574+0000","last_became_active":"2023-12-04T14:06:25.388163+0000","last_became_peered":"2023-12-04T14:06:25.388163+0000","last_unstale":"2023-12-04T14:06:26.744574+0000","last_undegraded":"2023-12-04T14:06:26.744574+0000","last_fullsized":"2023-12-04T14:06:26.744574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:17:26.277565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:26.744570+0000","last_change":"2023-12-04T14:06:25.390563+0000","last_active":"2023-12-04T14:06:26.744570+0000","last_peered":"2023-12-04T14:06:26.744570+0000","last_clean":"2023-12-04T14:06:26.744570+0000","last_became_active":"2023-12-04T14:06:25.390081+0000","last_became_peered":"2023-12-04T14:06:25.390081+0000","last_unstale":"2023-12-04T14:06:26.744570+0000","last_undegraded":"2023-12-04T14:06:26.744570+0000","last_fullsized":"2023-12-04T14:06:26.744570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:37:41.613703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:26.744697+0000","last_change":"2023-12-04T14:06:25.390097+0000","last_active":"2023-12-04T14:06:26.744697+0000","last_peered":"2023-12-04T14:06:26.744697+0000","last_clean":"2023-12-04T14:06:26.744697+0000","last_became_active":"2023-12-04T14:06:25.389613+0000","last_became_peered":"2023-12-04T14:06:25.389613+0000","last_unstale":"2023-12-04T14:06:26.744697+0000","last_undegraded":"2023-12-04T14:06:26.744697+0000","last_fullsized":"2023-12-04T14:06:26.744697+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:39:08.473833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:26.744575+0000","last_change":"2023-12-04T14:06:25.388414+0000","last_active":"2023-12-04T14:06:26.744575+0000","last_peered":"2023-12-04T14:06:26.744575+0000","last_clean":"2023-12-04T14:06:26.744575+0000","last_became_active":"2023-12-04T14:06:25.388105+0000","last_became_peered":"2023-12-04T14:06:25.388105+0000","last_unstale":"2023-12-04T14:06:26.744575+0000","last_undegraded":"2023-12-04T14:06:26.744575+0000","last_fullsized":"2023-12-04T14:06:26.744575+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:51:17.379730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:26.386193+0000","last_change":"2023-12-04T14:06:25.388568+0000","last_active":"2023-12-04T14:06:26.386193+0000","last_peered":"2023-12-04T14:06:26.386193+0000","last_clean":"2023-12-04T14:06:26.386193+0000","last_became_active":"2023-12-04T14:06:25.387970+0000","last_became_peered":"2023-12-04T14:06:25.387970+0000","last_unstale":"2023-12-04T14:06:26.386193+0000","last_undegraded":"2023-12-04T14:06:26.386193+0000","last_fullsized":"2023-12-04T14:06:26.386193+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-06T01:29:31.446711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:27.244861+0000","last_change":"2023-12-04T14:06:25.388574+0000","last_active":"2023-12-04T14:06:27.244861+0000","last_peered":"2023-12-04T14:06:27.244861+0000","last_clean":"2023-12-04T14:06:27.244861+0000","last_became_active":"2023-12-04T14:06:25.388254+0000","last_became_peered":"2023-12-04T14:06:25.388254+0000","last_unstale":"2023-12-04T14:06:27.244861+0000","last_undegraded":"2023-12-04T14:06:27.244861+0000","last_fullsized":"2023-12-04T14:06:27.244861+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:40:08.774747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:27.244849+0000","last_change":"2023-12-04T14:06:25.389261+0000","last_active":"2023-12-04T14:06:27.244849+0000","last_peered":"2023-12-04T14:06:27.244849+0000","last_clean":"2023-12-04T14:06:27.244849+0000","last_became_active":"2023-12-04T14:06:25.388830+0000","last_became_peered":"2023-12-04T14:06:25.388830+0000","last_unstale":"2023-12-04T14:06:27.244849+0000","last_undegraded":"2023-12-04T14:06:27.244849+0000","last_fullsized":"2023-12-04T14:06:27.244849+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:19:33.194202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-04T14:06:26.744651+0000","last_change":"2023-12-04T14:06:25.388372+0000","last_active":"2023-12-04T14:06:26.744651+0000","last_peered":"2023-12-04T14:06:26.744651+0000","last_clean":"2023-12-04T14:06:26.744651+0000","last_became_active":"2023-12-04T14:06:25.388074+0000","last_became_peered":"2023-12-04T14:06:25.388074+0000","last_unstale":"2023-12-04T14:06:26.744651+0000","last_undegraded":"2023-12-04T14:06:26.744651+0000","last_fullsized":"2023-12-04T14:06:26.744651+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:24.370718+0000","last_clean_scrub_stamp":"2023-12-04T14:06:24.370718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:31:08.592828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-04T14:06:25.386130+0000","last_change":"2023-12-04T14:06:19.745903+0000","last_active":"2023-12-04T14:06:25.386130+0000","last_peered":"2023-12-04T14:06:25.386130+0000","last_clean":"2023-12-04T14:06:25.386130+0000","last_became_active":"2023-12-04T14:06:19.745538+0000","last_became_peered":"2023-12-04T14:06:19.745538+0000","last_unstale":"2023-12-04T14:06:25.386130+0000","last_undegraded":"2023-12-04T14:06:25.386130+0000","last_fullsized":"2023-12-04T14:06:25.386130+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T14:06:18.731113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T14:06:18.731113+0000","last_clean_scrub_stamp":"2023-12-04T14:06:18.731113+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T23:32:15.394300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":34809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T14:06:30.127 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-12-04T14:06:30.127 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-12-04T14:06:30.127 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-12-04T14:06:30.128 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-12-04T14:06:30.438 INFO:teuthology.orchestra.run.smithi176.stdout: 2023-12-04T14:06:30.438 INFO:teuthology.orchestra.run.smithi176.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-04T14:06:30.451 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-12-04T14:06:30.451 INFO:teuthology.run_tasks:Running task exec... 2023-12-04T14:06:30.461 INFO:teuthology.task.exec:Executing custom commands... 2023-12-04T14:06:30.461 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi176.front.sepia.ceph.com 2023-12-04T14:06:30.462 DEBUG:teuthology.orchestra.run.smithi176:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-04T14:06:30.817 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror] 2023-12-04T14:06:30.817 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCe3G1lzO7eCRAAqWjJXldeL0QRQcvlJOhL4g== 2023-12-04T14:06:30.817 INFO:teuthology.orchestra.run.smithi176.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-04T14:06:30.817 INFO:teuthology.orchestra.run.smithi176.stdout: caps osd = "profile rbd" 2023-12-04T14:06:30.817 INFO:teuthology.orchestra.run.smithi176.stderr:updated caps for client.mirror 2023-12-04T14:06:30.830 DEBUG:teuthology.orchestra.run.smithi176:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-04T14:06:31.182 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.0] 2023-12-04T14:06:31.182 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCe3G1lqNZ/DRAANcSohWEy5uZMHA+P7AtxxQ== 2023-12-04T14:06:31.182 INFO:teuthology.orchestra.run.smithi176.stdout: caps mon = "profile rbd-mirror" 2023-12-04T14:06:31.183 INFO:teuthology.orchestra.run.smithi176.stdout: caps osd = "profile rbd" 2023-12-04T14:06:31.183 INFO:teuthology.orchestra.run.smithi176.stderr:updated caps for client.mirror.0 2023-12-04T14:06:31.196 DEBUG:teuthology.orchestra.run.smithi176:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-04T14:06:31.550 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.1] 2023-12-04T14:06:31.550 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCe3G1lpgXxEBAAffDZKuvnVR1SZBA/+x4K2w== 2023-12-04T14:06:31.550 INFO:teuthology.orchestra.run.smithi176.stdout: caps mon = "profile rbd-mirror" 2023-12-04T14:06:31.551 INFO:teuthology.orchestra.run.smithi176.stdout: caps osd = "profile rbd" 2023-12-04T14:06:31.551 INFO:teuthology.orchestra.run.smithi176.stderr:updated caps for client.mirror.1 2023-12-04T14:06:31.564 DEBUG:teuthology.orchestra.run.smithi176:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-04T14:06:31.917 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.2] 2023-12-04T14:06:31.917 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCe3G1lZg63FhAA5pNb4Gzqc4UFjkMqWXYM3A== 2023-12-04T14:06:31.917 INFO:teuthology.orchestra.run.smithi176.stdout: caps mon = "profile rbd-mirror" 2023-12-04T14:06:31.917 INFO:teuthology.orchestra.run.smithi176.stdout: caps osd = "profile rbd" 2023-12-04T14:06:31.917 INFO:teuthology.orchestra.run.smithi176.stderr:updated caps for client.mirror.2 2023-12-04T14:06:31.930 DEBUG:teuthology.orchestra.run.smithi176:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-04T14:06:32.284 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.3] 2023-12-04T14:06:32.284 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCe3G1lrNuCHBAA2FJPniiU5Peh7A1lC8H3ug== 2023-12-04T14:06:32.284 INFO:teuthology.orchestra.run.smithi176.stdout: caps mon = "profile rbd-mirror" 2023-12-04T14:06:32.284 INFO:teuthology.orchestra.run.smithi176.stdout: caps osd = "profile rbd" 2023-12-04T14:06:32.285 INFO:teuthology.orchestra.run.smithi176.stderr:updated caps for client.mirror.3 2023-12-04T14:06:32.297 DEBUG:teuthology.orchestra.run.smithi176:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-04T14:06:32.651 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror] 2023-12-04T14:06:32.652 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDC3G1lhBcTIRAA5Hf5VUuMOUloYfJliE0wVg== 2023-12-04T14:06:32.652 INFO:teuthology.orchestra.run.smithi176.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-04T14:06:32.652 INFO:teuthology.orchestra.run.smithi176.stdout: caps osd = "profile rbd" 2023-12-04T14:06:32.652 INFO:teuthology.orchestra.run.smithi176.stderr:updated caps for client.mirror 2023-12-04T14:06:32.665 DEBUG:teuthology.orchestra.run.smithi176:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-04T14:06:33.018 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.0] 2023-12-04T14:06:33.018 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDC3G1lm07SJhAAE0L2yQrvx0IVtCouka2f0g== 2023-12-04T14:06:33.018 INFO:teuthology.orchestra.run.smithi176.stdout: caps mon = "profile rbd-mirror" 2023-12-04T14:06:33.018 INFO:teuthology.orchestra.run.smithi176.stdout: caps osd = "profile rbd" 2023-12-04T14:06:33.019 INFO:teuthology.orchestra.run.smithi176.stderr:updated caps for client.mirror.0 2023-12-04T14:06:33.032 DEBUG:teuthology.orchestra.run.smithi176:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-04T14:06:33.387 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.1] 2023-12-04T14:06:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDC3G1lK9OkLBAA21OQ/ogYXHaFPWE8gVUT9g== 2023-12-04T14:06:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: caps mon = "profile rbd-mirror" 2023-12-04T14:06:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: caps osd = "profile rbd" 2023-12-04T14:06:33.387 INFO:teuthology.orchestra.run.smithi176.stderr:updated caps for client.mirror.1 2023-12-04T14:06:33.401 DEBUG:teuthology.orchestra.run.smithi176:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-04T14:06:33.753 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.2] 2023-12-04T14:06:33.754 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDC3G1lw31pMhAA9zzJiWK0A4y3viPOBfhD9g== 2023-12-04T14:06:33.754 INFO:teuthology.orchestra.run.smithi176.stdout: caps mon = "profile rbd-mirror" 2023-12-04T14:06:33.754 INFO:teuthology.orchestra.run.smithi176.stdout: caps osd = "profile rbd" 2023-12-04T14:06:33.754 INFO:teuthology.orchestra.run.smithi176.stderr:updated caps for client.mirror.2 2023-12-04T14:06:33.767 DEBUG:teuthology.orchestra.run.smithi176:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-04T14:06:34.121 INFO:teuthology.orchestra.run.smithi176.stdout:[client.mirror.3] 2023-12-04T14:06:34.121 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDC3G1liwEnOBAAPFhr39VCC6ujBmu05ZfwmQ== 2023-12-04T14:06:34.121 INFO:teuthology.orchestra.run.smithi176.stdout: caps mon = "profile rbd-mirror" 2023-12-04T14:06:34.121 INFO:teuthology.orchestra.run.smithi176.stdout: caps osd = "profile rbd" 2023-12-04T14:06:34.121 INFO:teuthology.orchestra.run.smithi176.stderr:updated caps for client.mirror.3 2023-12-04T14:06:34.134 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-04T14:06:34.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-04T14:06:34.145 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-04T14:06:34.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-04T14:06:34.147 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-04T14:06:34.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-04T14:06:34.156 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-04T14:06:34.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-04T14:06:34.158 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-04T14:06:34.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-04T14:06:34.167 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-04T14:06:34.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-04T14:06:34.169 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-04T14:06:34.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-04T14:06:34.178 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-04T14:06:34.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-04T14:06:34.181 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-04T14:06:34.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-04T14:06:34.189 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-04T14:06:34.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-04T14:06:34.191 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-04T14:06:34.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-04T14:06:34.201 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-04T14:06:34.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-04T14:06:34.203 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-04T14:06:34.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-04T14:06:34.212 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-04T14:06:34.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-04T14:06:34.215 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-04T14:06:34.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-04T14:06:34.224 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-04T14:06:34.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-04T14:06:34.228 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-04T14:06:34.238 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1701698794 2023-12-04T14:06:34.238 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-04T14:06:34.238 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-04T14:06:34.248 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1701698794 2023-12-04T14:06:34.248 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-04T14:06:34.248 INFO:teuthology.run_tasks:Running task workunit... 2023-12-04T14:06:34.260 INFO:tasks.workunit:Pulling workunits from ref 7c730b2996d0b00d56aea5ca791f90c534a13422 2023-12-04T14:06:34.260 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-04T14:06:34.260 DEBUG:teuthology.orchestra.run.smithi176:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-04T14:06:34.261 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-12-04T14:06:34.261 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-12-04T14:06:34.261 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-12-04T14:06:34.261 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-12-04T14:06:34.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T14:06:34.280 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-12-04T14:06:34.281 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-04T14:06:34.335 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-04T14:06:34.335 DEBUG:teuthology.orchestra.run.smithi176:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-12-04T14:06:34.389 INFO:tasks.workunit:timeout=6h 2023-12-04T14:06:34.389 INFO:tasks.workunit:cleanup=True 2023-12-04T14:06:34.389 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 7c730b2996d0b00d56aea5ca791f90c534a13422 2023-12-04T14:06:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-12-04T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Note: switching to '7c730b2996d0b00d56aea5ca791f90c534a13422'. 2023-12-04T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-04T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-04T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:state without impacting any branches by switching back to a branch. 2023-12-04T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-04T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-04T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: git switch -c 2023-12-04T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:07:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Or undo this operation with: 2023-12-04T14:07:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:07:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: git switch - 2023-12-04T14:07:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:07:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-04T14:07:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:07:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:HEAD is now at 7c730b2996d Revert "common: hide asio header from common/error_code.h" 2023-12-04T14:07:26.149 DEBUG:teuthology.orchestra.run.smithi176:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-12-04T14:07:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-04T14:07:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-04T14:07:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-04T14:07:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-04T14:07:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-04T14:07:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-04T14:07:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-04T14:07:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-04T14:07:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-04T14:07:27.168 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2023-12-04T14:07:27.168 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-12-04T14:07:27.223 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-12-04T14:07:27.224 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-12-04T14:07:27.224 DEBUG:teuthology.orchestra.run.smithi176:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c730b2996d0b00d56aea5ca791f90c534a13422 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-04T14:07:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ IMAGE_COUNT=50 2023-12-04T14:07:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ export LOCKDEP=0 2023-12-04T14:07:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ LOCKDEP=0 2023-12-04T14:07:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-04T14:07:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-12-04T14:07:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ type xmlstarlet 2023-12-04T14:07:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ XMLSTARLET=xmlstarlet 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ RBD_MIRROR_INSTANCES=4 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ CLUSTER1=cluster1 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ CLUSTER2=cluster2 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ PEER_CLUSTER_SUFFIX= 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ POOL=mirror 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ PARENT_POOL=mirror_parent 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ NS1=ns1 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ NS2=ns2 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ TEMPDIR= 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ CEPH_ID=mirror 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ RBD_IMAGE_FEATURES=layering 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ RBD_MIRROR_MODE=snapshot 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ MIRROR_POOL_MODE=image 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' snapshot = snapshot ']' 2023-12-04T14:07:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ MIRROR_POOL_MODE=image 2023-12-04T14:07:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ export 'CEPH_ARGS=--id mirror' 2023-12-04T14:07:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ CEPH_ARGS='--id mirror' 2023-12-04T14:07:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ LAST_MIRROR_INSTANCE=3 2023-12-04T14:07:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-04T14:07:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-12-04T14:07:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-12-04T14:07:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ CEPH_BIN=. 2023-12-04T14:07:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ CEPH_SRC=. 2023-12-04T14:07:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' -e CMakeCache.txt ']' 2023-12-04T14:07:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 -gt 0 ']' 2023-12-04T14:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ setup 2023-12-04T14:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local c 2023-12-04T14:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ trap 'cleanup $?' INT TERM EXIT 2023-12-04T14:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ setup_tempdir 2023-12-04T14:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:07:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mktemp -d 2023-12-04T14:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ TEMPDIR=/tmp/tmp.rDwoQpiP9p 2023-12-04T14:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -z 1 ']' 2023-12-04T14:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ setup_pools cluster1 cluster2 2023-12-04T14:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local remote_cluster=cluster2 2023-12-04T14:07:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mon_map_file 2023-12-04T14:07:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mon_addr 2023-12-04T14:07:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local admin_key_file 2023-12-04T14:07:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local uuid 2023-12-04T14:07:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:07:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2023-12-04T14:07:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:pool 'mirror' already exists 2023-12-04T14:07:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:07:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-12-04T14:07:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:pool 'mirror_parent' already exists 2023-12-04T14:07:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:07:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 pool init mirror 2023-12-04T14:07:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:07:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2023-12-04T14:07:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:07:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:07:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-12-04T14:07:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-12-04T14:07:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-12-04T14:07:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-12-04T14:07:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-12-04T14:07:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-12-04T14:07:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -z ']' 2023-12-04T14:07:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -z ']' 2023-12-04T14:07:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ peer_add cluster1 mirror cluster2 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local client_cluster=cluster2 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local remote_cluster=cluster2 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local uuid_var_name 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local error_code 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local peer_uuid 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 16 32 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:07:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-12-04T14:07:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ peer_uuid=4d244b58-7457-4b89-a2f7-d69add210e3b 2023-12-04T14:07:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=0 2023-12-04T14:07:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:07:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 0 -eq 17 ']' 2023-12-04T14:07:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 0 -eq 0 2023-12-04T14:07:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror_parent 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local client_cluster=cluster2 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local remote_cluster=cluster2 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local uuid_var_name 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local error_code 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local peer_uuid 2023-12-04T14:07:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 16 32 2023-12-04T14:07:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:07:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-12-04T14:07:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ peer_uuid=4d9db1f6-5ae6-4138-ab81-f317a659b2a2 2023-12-04T14:07:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=0 2023-12-04T14:07:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:07:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 0 -eq 17 ']' 2023-12-04T14:07:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 0 -eq 0 2023-12-04T14:07:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:07:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:07:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ setup_pools cluster2 cluster1 2023-12-04T14:07:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:07:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local remote_cluster=cluster1 2023-12-04T14:07:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mon_map_file 2023-12-04T14:07:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mon_addr 2023-12-04T14:07:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local admin_key_file 2023-12-04T14:07:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local uuid 2023-12-04T14:07:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:07:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2023-12-04T14:07:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:pool 'mirror' already exists 2023-12-04T14:07:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:07:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-12-04T14:07:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:pool 'mirror_parent' already exists 2023-12-04T14:07:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:07:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 pool init mirror 2023-12-04T14:07:38.262 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5975221346008389) > weight (0.25) 2023-12-04T14:07:38.262 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8069043634862697) > weight (0.25) 2023-12-04T14:07:38.262 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9658981895904757) > weight (0.25) 2023-12-04T14:07:38.262 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-04T14:07:38.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-04T14:07:38.263 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-04T14:07:38.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi176.stderr:2023-12-04T14:07:38.263+0000 7f4c436bf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 91073) UID: 1000 2023-12-04T14:07:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:07:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2023-12-04T14:07:43.264 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-04T14:07:43.265 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:07:43.265 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-04T14:07:43.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-04T14:07:43.265 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-04T14:07:43.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-04T14:07:43.269 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-12-04T14:07:43.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi176.stderr:2023-12-04T14:07:43.387+0000 7faa4bc1a640 -1 rbd::mirror::PoolReplayer: 0x555b70de8000 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-04T14:07:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:07:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:07:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-12-04T14:07:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-12-04T14:07:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-12-04T14:07:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-12-04T14:07:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-12-04T14:07:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-12-04T14:07:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -z ']' 2023-12-04T14:07:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -z ']' 2023-12-04T14:07:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ peer_add cluster2 mirror cluster1 2023-12-04T14:07:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:07:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:07:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local client_cluster=cluster1 2023-12-04T14:07:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local remote_cluster=cluster1 2023-12-04T14:07:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local uuid_var_name 2023-12-04T14:07:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:07:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local error_code 2023-12-04T14:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local peer_uuid 2023-12-04T14:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 16 32 2023-12-04T14:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-12-04T14:07:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ peer_uuid=5c2e1848-7634-4fdc-b12f-edf1a975873b 2023-12-04T14:07:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=0 2023-12-04T14:07:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:07:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 0 -eq 17 ']' 2023-12-04T14:07:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 0 -eq 0 2023-12-04T14:07:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:07:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:07:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-12-04T14:07:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:07:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror_parent 2023-12-04T14:07:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local client_cluster=cluster1 2023-12-04T14:07:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local remote_cluster=cluster1 2023-12-04T14:07:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local uuid_var_name 2023-12-04T14:07:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:07:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local error_code 2023-12-04T14:07:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local peer_uuid 2023-12-04T14:07:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 16 32 2023-12-04T14:07:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:07:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-12-04T14:07:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: mirror peer already exists 2023-12-04T14:07:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ peer_uuid= 2023-12-04T14:07:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=17 2023-12-04T14:07:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:07:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 17 -eq 17 ']' 2023-12-04T14:07:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:07:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd mirror pool info --cluster cluster2 --pool mirror_parent --format xml 2023-12-04T14:07:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//peers/peer[site_name='\''cluster1'\'']/uuid' 2023-12-04T14:07:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ peer_uuid=3406de70-179c-4307-8204-709dd488b6ef 2023-12-04T14:07:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:07:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 --pool mirror_parent mirror pool peer remove 3406de70-179c-4307-8204-709dd488b6ef 2023-12-04T14:07:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 16 32 2023-12-04T14:07:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:07:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-12-04T14:07:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ peer_uuid=95deac7d-1a0f-4786-8887-0a2528de7e7e 2023-12-04T14:07:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=0 2023-12-04T14:07:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:07:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 0 -eq 17 ']' 2023-12-04T14:07:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 0 -eq 0 2023-12-04T14:07:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:07:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:07:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:07:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ start_mirrors cluster1 2023-12-04T14:07:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:07:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 0 3 2023-12-04T14:07:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ start_mirror cluster1:0 2023-12-04T14:07:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1:0 2023-12-04T14:07:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance 2023-12-04T14:07:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:07:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local val=cluster1:0 2023-12-04T14:07:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster_var_name=cluster 2023-12-04T14:07:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance_var_name=instance 2023-12-04T14:07:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cluster=cluster1 2023-12-04T14:07:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ instance=0 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 0 = cluster1:0 ']' 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval cluster=cluster1 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval instance=0 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ start_mirror cluster1:1 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1:1 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local val=cluster1:1 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster_var_name=cluster 2023-12-04T14:07:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance_var_name=instance 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cluster=cluster1 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ instance=1 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 1 = cluster1:1 ']' 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval cluster=cluster1 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval instance=1 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ start_mirror cluster1:2 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1:2 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local val=cluster1:2 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster_var_name=cluster 2023-12-04T14:07:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance_var_name=instance 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cluster=cluster1 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ instance=2 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 2 = cluster1:2 ']' 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval cluster=cluster1 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval instance=2 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ start_mirror cluster1:3 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1:3 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local val=cluster1:3 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster_var_name=cluster 2023-12-04T14:07:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance_var_name=instance 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cluster=cluster1 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ instance=3 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 3 = cluster1:3 ']' 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval cluster=cluster1 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval instance=3 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ start_mirrors cluster2 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 0 3 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ start_mirror cluster2:0 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2:0 2023-12-04T14:07:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local val=cluster2:0 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster_var_name=cluster 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance_var_name=instance 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cluster=cluster2 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ instance=0 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 0 = cluster2:0 ']' 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval cluster=cluster2 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster2 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval instance=0 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ start_mirror cluster2:1 2023-12-04T14:07:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2:1 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local val=cluster2:1 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster_var_name=cluster 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance_var_name=instance 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cluster=cluster2 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ instance=1 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 1 = cluster2:1 ']' 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval cluster=cluster2 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster2 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval instance=1 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ start_mirror cluster2:2 2023-12-04T14:07:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2:2 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local val=cluster2:2 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster_var_name=cluster 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance_var_name=instance 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cluster=cluster2 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ instance=2 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 2 = cluster2:2 ']' 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval cluster=cluster2 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster2 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval instance=2 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ start_mirror cluster2:3 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2:3 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local val=cluster2:3 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster_var_name=cluster 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local instance_var_name=instance 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cluster=cluster2 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ instance=3 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 3 = cluster2:3 ']' 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval cluster=cluster2 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster2 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ eval instance=3 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T14:07:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee -a /tmp/tmp.rDwoQpiP9p/rbd-mirror.test.log 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ date '+%F %T' 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04 14:07:44 TEST: add image and test replay after client crashes 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:2023-12-04 14:07:44 TEST: add image and test replay after client crashes 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=test 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:07:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror test 512M 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 512M ']' 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=512M 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:07:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/test 2023-12-04T14:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-12-04T14:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:07:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-12-04T14:07:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:07:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-12-04T14:07:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'test': 2023-12-04T14:07:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 512 MiB in 128 objects 2023-12-04T14:07:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:07:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:07:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10b8e49ad067 2023-12-04T14:07:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10b8e49ad067 2023-12-04T14:07:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:07:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:07:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:07:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:07:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:07:44 2023 2023-12-04T14:07:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:07:44 2023 2023-12-04T14:07:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:07:44 2023 2023-12-04T14:07:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:07:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:07:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:07:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:07:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-04T14:07:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:07:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:07:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:07:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-04T14:07:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:07:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:07:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:07:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:07:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:07:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:07:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:07:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-04T14:07:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:07:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:07:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:07:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:07:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-04T14:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:07:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:07:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:07:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:07:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:07:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:07:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:07:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:07:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:07:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:07:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:07:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:07:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:07:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:07:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:07:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:07:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:07:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:07:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:07:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:07:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:07:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:07:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:07:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:07:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:07:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:07:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:07:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:07:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:07:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:07:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:07:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:07:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:07:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:07:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:07:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:07:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:07:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:07:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:07:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:07:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:07:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:07:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:07:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:07:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:07:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:07:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:07:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:07:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:07:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:07:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:07:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:07:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:07:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:07:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:07:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:07:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:07:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:07:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-04T14:07:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:07:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:07:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:07:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:07:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:07:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:07:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:07:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:07:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:07:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:07:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:07:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:07:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:07:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:07:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:07:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:07:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:07:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:07:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:07:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-04T14:07:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:07:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 1 2023-12-04T14:07:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:07:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:07:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:07:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:07:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:07:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:07:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:07:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:07:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:07:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:07:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:07:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:07:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:07:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:07:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:07:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:07:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:07:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:07:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:07:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:07:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:07:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:07:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:07:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:07:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:07:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:07:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:07:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:07:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:07:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:07:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:07:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:07:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:07:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:07:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:07:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:07:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:07:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:07:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:07:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:07:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:07:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:07:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:07:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:07:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:07:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:07:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:07:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:07:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:07:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:07:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:07:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:07:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:07:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:07:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:07:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:07:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:07:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:07:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:07:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:07:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:07:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:07:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:07:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-04T14:07:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:07:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:07:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:07:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:07:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:07:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:07:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:07:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:07:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:07:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:07:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:07:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:07:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:07:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:07:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:07:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:07:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:07:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:07:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:07:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-04T14:07:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:07:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 2 2023-12-04T14:07:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:07:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:07:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:07:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:07:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:07:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:07:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:07:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:07:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:07:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:07:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:07:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:07:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:07:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:07:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:07:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:07:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:07:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:07:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:07:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:07:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:07:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:07:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:07:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:07:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:07:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:07:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:07:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:07:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:07:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:07:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:07:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:07:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:07:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:07:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:07:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:07:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:07:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:07:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:07:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:07:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:07:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:07:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:07:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:07:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:07:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:07:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:07:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:07:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:07:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:07:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:07:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:07:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:07:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:07:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:07:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:07:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:07:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:07:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-04T14:07:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:07:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 4 2023-12-04T14:07:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:07:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:07:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:07:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:07:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:07:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:07:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:07:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:07:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:07:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:07:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:07:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:07:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:07:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:07:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:07:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:07:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:07:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:07:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:07:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:07:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:07:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:07:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:07:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:07:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:07:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:07:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:07:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:07:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:07:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:07:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:07:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:07:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:07:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:07:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:07:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:07:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-04T14:07:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:07:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:07:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:07:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:07:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 8 2023-12-04T14:08:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:08:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:08:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:08:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:08:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:08:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:08:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:08:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:08:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:08:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:08:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:08:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:08:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:08:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:08:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:08:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:08:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:08:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:08:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:08:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:08:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:08:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:08:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:08:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:08:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:08:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:08:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:08:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:08:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:08:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:08:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:08:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:08:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:08:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:08:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:08:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:08:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:08:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:08:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:08:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:08:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:08:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:08:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:08:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:08:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:08:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:08:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:08:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:08:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-04T14:08:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:08:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:08:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:08:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:08:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 8 2023-12-04T14:08:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:08:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:08:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:08:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:08:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:08:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:08:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:08:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:08:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:08:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:08:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:08:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:08:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:08:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:08:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:08:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:08:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:08:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:08:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:08:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:08:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:08:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:08:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:08:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:08:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:08:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:08:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:08:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:08:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:08:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:08:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:08:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:08:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:08:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:08:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:08:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:08:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:08:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:08:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:08:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:08:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:08:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:08:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:08:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:08:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:08:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:08:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:08:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:08:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:08:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:08:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:08:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:08:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:08:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:08:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:08:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:08:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:08:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:08:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:08:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:08:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:08:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:08:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:08:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:08:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:08:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:08:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:08:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:08:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:08:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:08:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:08:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:08:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:08:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:08:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:08:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:08:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:08:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:08:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:08:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:08:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:08:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:08:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:08:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:08:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:08:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:08:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:08:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:08:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:08:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:08:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ clean_snap_name= 2023-12-04T14:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 1 10 2023-12-04T14:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror test 2023-12-04T14:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:08:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=5 2023-12-04T14:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:08:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-04T14:08:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 64194560 bytes 2023-12-04T14:08:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 158496768 bytes 2023-12-04T14:08:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 213450752 bytes 2023-12-04T14:08:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 240892928 bytes 2023-12-04T14:08:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 299832320 bytes 2023-12-04T14:08:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 360512512 bytes 2023-12-04T14:08:13.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi176.stderr:2023-12-04T14:08:13.649+0000 7faa4bc1a640 -1 rbd::mirror::Mirror: 0x555b6ffd3d40 update_pool_replayers: restarting failed pool replayer for uuid: 4d244b58-7457-4b89-a2f7-d69add210e3b cluster: cluster2 client: client.mirror 2023-12-04T14:08:13.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi176.stderr:2023-12-04T14:08:13.666+0000 7faa2dd41640 -1 rbd::mirror::PoolReplayer: 0x555b70ea9b00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-12-04T14:08:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 419159040 bytes 2023-12-04T14:08:13.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi176.stderr:2023-12-04T14:08:13.732+0000 7faa4bc1a640 -1 rbd::mirror::Mirror: 0x555b6ffd3d40 update_pool_replayers: restarting failed pool replayer for uuid: 4d9db1f6-5ae6-4138-ab81-f317a659b2a2 cluster: cluster2 client: client.mirror 2023-12-04T14:08:13.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 453108736 bytes 2023-12-04T14:08:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 465451008 bytes 2023-12-04T14:08:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 490335232 bytes 2023-12-04T14:08:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 509036544 bytes 2023-12-04T14:08:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 529789952 bytes 2023-12-04T14:08:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 603398144 bytes 2023-12-04T14:08:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 613047296 bytes 2023-12-04T14:08:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 634485760 bytes 2023-12-04T14:08:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 705529856 bytes 2023-12-04T14:08:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 748559360 bytes 2023-12-04T14:08:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 785079296 bytes 2023-12-04T14:08:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 821962752 bytes 2023-12-04T14:08:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 822571008 bytes 2023-12-04T14:08:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 841360384 bytes 2023-12-04T14:08:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 888689664 bytes 2023-12-04T14:08:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 958109696 bytes 2023-12-04T14:08:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1006364672 bytes 2023-12-04T14:08:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1113284608 bytes 2023-12-04T14:08:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1125735424 bytes 2023-12-04T14:08:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1126366208 bytes 2023-12-04T14:08:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1195117568 bytes 2023-12-04T14:08:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1235111936 bytes 2023-12-04T14:08:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1256339456 bytes 2023-12-04T14:08:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1287069696 bytes 2023-12-04T14:08:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-04T14:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:08:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:08:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:08:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:08:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:08:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:08:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:08:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:08:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:08:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:08:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:08:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:08:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:08:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: down+unknown 2023-12-04T14:08:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: status not found 2023-12-04T14:08:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04T14:08:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:08:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:08:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: down+unknown 2023-12-04T14:08:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: status not found 2023-12-04T14:08:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04T14:08:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 1 2023-12-04T14:08:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:08:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 2 2023-12-04T14:08:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:08:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:08:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:08:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:08:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:08:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:08:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:08:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:08:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:08:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:08:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:08:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: down+unknown 2023-12-04T14:08:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: status not found 2023-12-04T14:08:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04T14:08:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:08:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:08:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: down+unknown 2023-12-04T14:08:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: status not found 2023-12-04T14:08:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04T14:08:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 1 2023-12-04T14:08:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:08:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 4 2023-12-04T14:08:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:08:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:08:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:08:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:08:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:08:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:08:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:08:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:08:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:08:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:24.262 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6377122116891564) > weight (0.25) 2023-12-04T14:08:24.262 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3473183921297902) > weight (0.25) 2023-12-04T14:08:24.262 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.42734444971755525) > weight (0.25) 2023-12-04T14:08:24.263 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6561190330746931) > weight (0.25) 2023-12-04T14:08:24.263 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-12-04T14:08:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:08:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:08:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: down+unknown 2023-12-04T14:08:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: status not found 2023-12-04T14:08:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04T14:08:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:08:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:08:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: down+unknown 2023-12-04T14:08:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: status not found 2023-12-04T14:08:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04T14:08:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 1 2023-12-04T14:08:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:08:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 8 2023-12-04T14:08:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:08:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:08:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:08:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:08:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:08:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:08:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: down+unknown 2023-12-04T14:08:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: status not found 2023-12-04T14:08:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04T14:08:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:08:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:08:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: down+unknown 2023-12-04T14:08:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: status not found 2023-12-04T14:08:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04T14:08:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 1 2023-12-04T14:08:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:08:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 8 2023-12-04T14:08:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:08:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:08:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:08:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:08:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:08:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:08:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:08:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:08:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:08:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:08:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:08:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+replaying 2023-12-04T14:08:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701698865,"remote_snapshot_timestamp":1701698865,"replay_state":"idle"} 2023-12-04T14:08:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: service: mirror.2 on smithi176 2023-12-04T14:08:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:08:34 2023-12-04T14:08:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:08:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:08:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+stopped 2023-12-04T14:08:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: local image is primary 2023-12-04T14:08:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:08:34 2023-12-04T14:08:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:08:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ description: .*' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701698865,"remote_snapshot_timestamp":1701698865,"replay_state":"idle"} 2023-12-04T14:08:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:08:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:08:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^up+' 2023-12-04T14:08:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:08:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.2 on smithi176 2023-12-04T14:08:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:08:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'last_update=2023-12-04 14:08:34' 2023-12-04T14:08:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-04 14:08:34' 2023-12-04T14:08:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:08:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'prev_last_update=2023-12-04 14:08:34' 2023-12-04T14:08:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-04T14:08:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror.mirror_status 2023-12-04T14:08:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:08:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror.mirror_status 2023-12-04T14:08:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-04T14:08:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:08:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local last_update state 2023-12-04T14:08:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-04T14:08:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ last_update='2023-12-04 14:08:34' 2023-12-04T14:08:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-04T14:08:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=up+replaying 2023-12-04T14:08:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:08:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep up+replaying 2023-12-04T14:08:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:08:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:08:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:08:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap1 2023-12-04T14:08:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap1 2023-12-04T14:08:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:08:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap1 2023-12-04T14:08:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:08:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-04T14:08:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-04T14:08:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:08:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:08:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:08:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:08:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-04T14:08:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:08:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:08:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:08:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-04T14:08:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:08:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:08:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:08:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:08:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:08:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:08:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:08:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:08:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:08:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:08:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:08:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:08:48.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:08:48.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:08:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:08:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:08:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:08:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:08:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:08:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:08:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:08:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:08:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:08:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:08:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:08:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:08:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:08:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:08:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:08:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:08:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:08:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:08:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:08:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:08:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:08:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:08:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:08:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:08:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:08:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:08:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:08:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:08:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:08:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:08:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:08:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:08:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:08:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:08:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:08:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:08:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:08:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:08:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-04T14:08:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:08:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:08:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:08:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:08:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-04T14:08:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:08:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:08:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-test.status 2023-12-04T14:08:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-test.status 2023-12-04T14:08:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:08:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:08:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.status 2023-12-04T14:08:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.status 2023-12-04T14:08:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:08:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:08:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-04T14:08:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:08:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-04T14:08:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 6 2023-12-04T14:08:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:08:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:08:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:08:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-04T14:08:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:08:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:08:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=6 2023-12-04T14:08:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:08:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:08:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:08:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:08:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:08:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:08:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:08:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:08:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=4 2023-12-04T14:08:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 6 = 4 2023-12-04T14:08:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:08:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:08:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:08:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:08:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:08:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:08:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:08:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:08:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:08:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:08:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:08:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:08:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 6 = '' 2023-12-04T14:08:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:08:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:08:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:08:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:08:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:08:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:08:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:08:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:08:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:08:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:08:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:08:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:08:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 6 = '' 2023-12-04T14:08:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:08:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:08:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:08:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:08:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:08:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:08:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:08:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=6 2023-12-04T14:08:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 6 = 6 2023-12-04T14:08:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:08:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-12-04T14:08:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:08:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap1 2023-12-04T14:08:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:08:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:08:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:08:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-12-04T14:08:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'test': 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 512 MiB in 128 objects 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10ad877be9cc 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10ad877be9cc 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:08:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:08:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:08:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:08:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:08:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:08:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:08:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:08:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap1 2023-12-04T14:08:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap1 2023-12-04T14:08:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:08:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:08:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:08:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:08:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:08:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:08:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:08:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:08:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:08:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:08:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:08:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ clean_snap_name=snap1-clean 2023-12-04T14:08:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-12-04T14:08:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:08:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap1-clean 2023-12-04T14:08:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-12-04T14:08:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:08:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror test 2023-12-04T14:08:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:08:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:08:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:08:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:08:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=14 2023-12-04T14:08:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:08:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-04T14:08:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 36023296 bytes 2023-12-04T14:08:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 97555456 bytes 2023-12-04T14:08:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 120829952 bytes 2023-12-04T14:08:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 183439360 bytes 2023-12-04T14:08:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 205888512 bytes 2023-12-04T14:08:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 280183808 bytes 2023-12-04T14:08:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 322582528 bytes 2023-12-04T14:08:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 349316096 bytes 2023-12-04T14:08:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 448129024 bytes 2023-12-04T14:08:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 507832320 bytes 2023-12-04T14:08:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 527654912 bytes 2023-12-04T14:08:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 544948224 bytes 2023-12-04T14:08:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 580901888 bytes 2023-12-04T14:08:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 624709632 bytes 2023-12-04T14:08:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 685300736 bytes 2023-12-04T14:08:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 745029632 bytes 2023-12-04T14:08:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 778540032 bytes 2023-12-04T14:08:58.270 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.703036180071263) > weight (0.25) 2023-12-04T14:08:58.270 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-04T14:08:58.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-04T14:08:58.271 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-04T14:08:58.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi176.stderr:2023-12-04T14:08:58.270+0000 7f2dc532a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 91039) UID: 1000 2023-12-04T14:08:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 854032384 bytes 2023-12-04T14:08:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 893097984 bytes 2023-12-04T14:08:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 991155200 bytes 2023-12-04T14:08:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 1052533760 bytes 2023-12-04T14:08:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1065247744 bytes 2023-12-04T14:08:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1076810752 bytes 2023-12-04T14:08:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1148010496 bytes 2023-12-04T14:08:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1186312192 bytes 2023-12-04T14:08:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1217281024 bytes 2023-12-04T14:08:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1250872320 bytes 2023-12-04T14:08:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1304485888 bytes 2023-12-04T14:09:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1398575104 bytes 2023-12-04T14:09:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1422494720 bytes 2023-12-04T14:09:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1435655168 bytes 2023-12-04T14:09:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1480557568 bytes 2023-12-04T14:09:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1501312000 bytes 2023-12-04T14:09:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1587947520 bytes 2023-12-04T14:09:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1588547584 bytes 2023-12-04T14:09:01.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1643865088 bytes 2023-12-04T14:09:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1768903680 bytes 2023-12-04T14:09:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1812230144 bytes 2023-12-04T14:09:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1859042304 bytes 2023-12-04T14:09:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1908814848 bytes 2023-12-04T14:09:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1928741888 bytes 2023-12-04T14:09:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1966708736 bytes 2023-12-04T14:09:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 2028898304 bytes 2023-12-04T14:09:02.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 2081184768 bytes 2023-12-04T14:09:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 2154558464 bytes 2023-12-04T14:09:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 2220381184 bytes 2023-12-04T14:09:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2269740032 bytes 2023-12-04T14:09:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2306099200 bytes 2023-12-04T14:09:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2370855936 bytes 2023-12-04T14:09:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2433671168 bytes 2023-12-04T14:09:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2505500672 bytes 2023-12-04T14:09:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2556582912 bytes 2023-12-04T14:09:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2566607872 bytes 2023-12-04T14:09:04.272 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-04T14:09:04.272 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:09:04.272 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-04T14:09:04.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-04T14:09:04.272 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-04T14:09:04.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-04T14:09:04.276 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2023-12-04T14:09:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2621488128 bytes 2023-12-04T14:09:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2696272896 bytes 2023-12-04T14:09:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2757686272 bytes 2023-12-04T14:09:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2788372480 bytes 2023-12-04T14:09:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2836924416 bytes 2023-12-04T14:09:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2922234880 bytes 2023-12-04T14:09:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2954926080 bytes 2023-12-04T14:09:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2990893056 bytes 2023-12-04T14:09:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 3042015232 bytes 2023-12-04T14:09:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 3099156480 bytes 2023-12-04T14:09:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 3138733056 bytes 2023-12-04T14:09:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 3197515776 bytes 2023-12-04T14:09:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 3245595648 bytes 2023-12-04T14:09:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 3309962240 bytes 2023-12-04T14:09:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 3424214016 bytes 2023-12-04T14:09:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 3470833664 bytes 2023-12-04T14:09:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 3508825088 bytes 2023-12-04T14:09:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 3552557056 bytes 2023-12-04T14:09:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 3577871360 bytes 2023-12-04T14:09:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7300 IOs, 3602464768 bytes 2023-12-04T14:09:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7400 IOs, 3676995584 bytes 2023-12-04T14:09:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7500 IOs, 3722763264 bytes 2023-12-04T14:09:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7600 IOs, 3733742592 bytes 2023-12-04T14:09:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7700 IOs, 3751214080 bytes 2023-12-04T14:09:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7800 IOs, 3776932864 bytes 2023-12-04T14:09:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7900 IOs, 3837809664 bytes 2023-12-04T14:09:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8000 IOs, 3928069120 bytes 2023-12-04T14:09:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8100 IOs, 3994857472 bytes 2023-12-04T14:09:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8200 IOs, 4028161024 bytes 2023-12-04T14:09:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8300 IOs, 4060783616 bytes 2023-12-04T14:09:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:09:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:09:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:09:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:09:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-04T14:09:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:09:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:09:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:09:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:09:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:09:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:09:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:09:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:09:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:09:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:09:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:09:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:09:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:09:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:09:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:09:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:09:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:09:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+replaying 2023-12-04T14:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: replaying, {"bytes_per_second":16315221.33,"bytes_per_snapshot":244728320.0,"last_snapshot_bytes":489456640,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701698929,"remote_snapshot_timestamp":1701698929,"replay_state":"idle"} 2023-12-04T14:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: service: mirror.2 on smithi176 2023-12-04T14:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:09:04 2023-12-04T14:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:09:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+stopped 2023-12-04T14:09:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: local image is primary 2023-12-04T14:09:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:09:04 2023-12-04T14:09:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:09:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ description: .*' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:09:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":16315221.33,"bytes_per_snapshot":244728320.0,"last_snapshot_bytes":489456640,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701698929,"remote_snapshot_timestamp":1701698929,"replay_state":"idle"} 2023-12-04T14:09:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:09:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:09:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^up+' 2023-12-04T14:09:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:09:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:09:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.2 on smithi176 2023-12-04T14:09:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:09:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'last_update=2023-12-04 14:09:04' 2023-12-04T14:09:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-04 14:09:04' 2023-12-04T14:09:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:09:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:09:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'prev_last_update=2023-12-04 14:09:04' 2023-12-04T14:09:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-04T14:09:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror.mirror_status 2023-12-04T14:09:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:09:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror.mirror_status 2023-12-04T14:09:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-04T14:09:10.551 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:09:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local last_update state 2023-12-04T14:09:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-04T14:09:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ last_update='2023-12-04 14:09:04' 2023-12-04T14:09:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-04T14:09:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=up+replaying 2023-12-04T14:09:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:09:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep up+replaying 2023-12-04T14:09:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:09:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:09:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:09:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap2 2023-12-04T14:09:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap2 2023-12-04T14:09:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:09:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap2 2023-12-04T14:09:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:09:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-04T14:09:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:09:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-04T14:09:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:09:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:09:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:09:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:09:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:09:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-04T14:09:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:09:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:09:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:09:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:09:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-04T14:09:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:09:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:09:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:09:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:09:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:09:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:09:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:09:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:09:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:09:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:09:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:09:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:09:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:09:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:09:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:09:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:09:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:09:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:09:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:09:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:09:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:09:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:09:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:09:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:09:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:09:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:09:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:09:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:09:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:09:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:09:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:09:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:09:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:09:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:09:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:09:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:09:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:09:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:09:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:09:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:09:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:09:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:09:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:09:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:09:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:09:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:09:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:09:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:09:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:09:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:09:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:09:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:09:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:09:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:09:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:09:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:09:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:09:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:09:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:09:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:09:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:09:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:09:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:09:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:09:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:09:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:09:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:09:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:09:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:09:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:09:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:09:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:09:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:09:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:09:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:09:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:09:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:09:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-04T14:09:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:09:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:09:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:09:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:09:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-04T14:09:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:09:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:09:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-test.status 2023-12-04T14:09:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-test.status 2023-12-04T14:09:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:09:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:09:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.status 2023-12-04T14:09:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.status 2023-12-04T14:09:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:09:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:09:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-04T14:09:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:09:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-04T14:09:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 9 2023-12-04T14:09:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:09:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:09:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:09:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-04T14:09:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:09:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:09:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=9 2023-12-04T14:09:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:09:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:09:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:09:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:09:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:09:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:09:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:09:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:09:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:09:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:09:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:09:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:09:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:09:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:09:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 9 = '' 2023-12-04T14:09:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:09:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:09:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:09:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:09:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:09:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:09:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:09:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:09:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:09:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:09:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:09:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:09:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:09:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 9 = '' 2023-12-04T14:09:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:09:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:09:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:09:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:09:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:09:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:09:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:09:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:09:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:09:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:09:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:09:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:09:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:09:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 9 = '' 2023-12-04T14:09:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:09:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:09:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:09:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:09:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:09:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:09:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:09:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:09:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=9 2023-12-04T14:09:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 9 = 9 2023-12-04T14:09:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:09:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-12-04T14:09:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:09:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap2 2023-12-04T14:09:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:09:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:09:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:09:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-12-04T14:09:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'test': 2023-12-04T14:09:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 512 MiB in 128 objects 2023-12-04T14:09:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:09:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 4 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10ad877be9cc 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10ad877be9cc 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:09:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:09:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:09:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snap1-clean ']' 2023-12-04T14:09:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-04T14:09:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap1-clean 2023-12-04T14:09:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:09:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:09:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:09:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:09:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:09:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:09:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:09:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:09:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:09:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:09:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:09:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap2 2023-12-04T14:09:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap2 2023-12-04T14:09:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:09:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:09:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:09:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:09:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:09:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:09:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:09:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:09:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:09:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:09:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:09:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ clean_snap_name=snap2-clean 2023-12-04T14:09:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-12-04T14:09:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:09:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap2-clean 2023-12-04T14:09:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-12-04T14:09:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:09:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror test 2023-12-04T14:09:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:09:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:09:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:09:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:09:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=14 2023-12-04T14:09:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:09:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-04T14:09:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 45367296 bytes 2023-12-04T14:09:49.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 51013632 bytes 2023-12-04T14:09:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 102882304 bytes 2023-12-04T14:09:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 149700608 bytes 2023-12-04T14:09:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 197787648 bytes 2023-12-04T14:09:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 253431808 bytes 2023-12-04T14:09:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 317647872 bytes 2023-12-04T14:09:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 378100736 bytes 2023-12-04T14:09:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 378735616 bytes 2023-12-04T14:09:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 430142464 bytes 2023-12-04T14:09:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 441364480 bytes 2023-12-04T14:09:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 480873472 bytes 2023-12-04T14:09:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 539330560 bytes 2023-12-04T14:09:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 570819584 bytes 2023-12-04T14:09:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 620404736 bytes 2023-12-04T14:09:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 642813952 bytes 2023-12-04T14:09:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 651642880 bytes 2023-12-04T14:09:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 652261376 bytes 2023-12-04T14:09:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 733341696 bytes 2023-12-04T14:09:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 807590912 bytes 2023-12-04T14:09:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 880318464 bytes 2023-12-04T14:09:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 940404736 bytes 2023-12-04T14:09:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 991341568 bytes 2023-12-04T14:09:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1015017472 bytes 2023-12-04T14:09:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1048635392 bytes 2023-12-04T14:09:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1085779968 bytes 2023-12-04T14:09:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1139659776 bytes 2023-12-04T14:09:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1153448960 bytes 2023-12-04T14:09:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1217725440 bytes 2023-12-04T14:09:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1232345088 bytes 2023-12-04T14:09:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1307939840 bytes 2023-12-04T14:09:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1335744512 bytes 2023-12-04T14:09:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1383343104 bytes 2023-12-04T14:09:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1480470528 bytes 2023-12-04T14:09:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1518210048 bytes 2023-12-04T14:09:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1562418176 bytes 2023-12-04T14:09:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1564389376 bytes 2023-12-04T14:09:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1587442688 bytes 2023-12-04T14:09:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1666314240 bytes 2023-12-04T14:09:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1706795008 bytes 2023-12-04T14:09:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1739747328 bytes 2023-12-04T14:09:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1763821568 bytes 2023-12-04T14:09:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1768709120 bytes 2023-12-04T14:09:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1808660480 bytes 2023-12-04T14:09:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1846114304 bytes 2023-12-04T14:09:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1884856320 bytes 2023-12-04T14:09:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1924439040 bytes 2023-12-04T14:09:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1960157184 bytes 2023-12-04T14:09:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2013722624 bytes 2023-12-04T14:09:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2038119424 bytes 2023-12-04T14:09:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2064870400 bytes 2023-12-04T14:09:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2142702592 bytes 2023-12-04T14:09:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2185287680 bytes 2023-12-04T14:09:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2192496640 bytes 2023-12-04T14:09:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2203866112 bytes 2023-12-04T14:09:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2272914432 bytes 2023-12-04T14:09:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2298654720 bytes 2023-12-04T14:09:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2306243584 bytes 2023-12-04T14:09:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2341002240 bytes 2023-12-04T14:09:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2388776960 bytes 2023-12-04T14:09:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2466919424 bytes 2023-12-04T14:09:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2506183680 bytes 2023-12-04T14:09:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2580092928 bytes 2023-12-04T14:09:59.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2655760384 bytes 2023-12-04T14:09:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2717314048 bytes 2023-12-04T14:09:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2733290496 bytes 2023-12-04T14:09:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2758341632 bytes 2023-12-04T14:10:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2785624064 bytes 2023-12-04T14:10:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2831734784 bytes 2023-12-04T14:10:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 2837038080 bytes 2023-12-04T14:10:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 2837636096 bytes 2023-12-04T14:10:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 2883608576 bytes 2023-12-04T14:10:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7300 IOs, 2943642624 bytes 2023-12-04T14:10:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7400 IOs, 2985216000 bytes 2023-12-04T14:10:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7500 IOs, 3035413504 bytes 2023-12-04T14:10:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7600 IOs, 3087156224 bytes 2023-12-04T14:10:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7700 IOs, 3130603520 bytes 2023-12-04T14:10:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7800 IOs, 3203481600 bytes 2023-12-04T14:10:01.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7900 IOs, 3243043840 bytes 2023-12-04T14:10:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8000 IOs, 3288387584 bytes 2023-12-04T14:10:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8100 IOs, 3326528512 bytes 2023-12-04T14:10:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8200 IOs, 3343893504 bytes 2023-12-04T14:10:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8300 IOs, 3417916416 bytes 2023-12-04T14:10:02.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8400 IOs, 3418521600 bytes 2023-12-04T14:10:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8500 IOs, 3463999488 bytes 2023-12-04T14:10:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8600 IOs, 3468593152 bytes 2023-12-04T14:10:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8700 IOs, 3508252672 bytes 2023-12-04T14:10:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8800 IOs, 3537890304 bytes 2023-12-04T14:10:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8900 IOs, 3588639744 bytes 2023-12-04T14:10:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:9000 IOs, 3589260288 bytes 2023-12-04T14:10:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:9100 IOs, 3612894208 bytes 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:10:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:10:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:10:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:10:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:10:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:10:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:10:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:10:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:10:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:10:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:10:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:10:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:10:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:10:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:10:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+replaying 2023-12-04T14:10:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: replaying, {"bytes_per_second":17819682.13,"bytes_per_snapshot":512023552.0,"last_snapshot_bytes":534590464,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701698980,"remote_snapshot_timestamp":1701698980,"replay_state":"idle"} 2023-12-04T14:10:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: service: mirror.2 on smithi176 2023-12-04T14:10:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:10:04 2023-12-04T14:10:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:10:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:10:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+stopped 2023-12-04T14:10:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: local image is primary 2023-12-04T14:10:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:10:04 2023-12-04T14:10:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:10:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:10:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ description: .*' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:10:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":17819682.13,"bytes_per_snapshot":512023552.0,"last_snapshot_bytes":534590464,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701698980,"remote_snapshot_timestamp":1701698980,"replay_state":"idle"} 2023-12-04T14:10:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:10:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:10:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^up+' 2023-12-04T14:10:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:10:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:10:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.2 on smithi176 2023-12-04T14:10:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:10:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'last_update=2023-12-04 14:10:04' 2023-12-04T14:10:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-04 14:10:04' 2023-12-04T14:10:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:10:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'prev_last_update=2023-12-04 14:10:04' 2023-12-04T14:10:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-04T14:10:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror.mirror_status 2023-12-04T14:10:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:10:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror.mirror_status 2023-12-04T14:10:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-04T14:10:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:10:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local last_update state 2023-12-04T14:10:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-04T14:10:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ last_update='2023-12-04 14:10:04' 2023-12-04T14:10:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-04T14:10:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=up+replaying 2023-12-04T14:10:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:10:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep up+replaying 2023-12-04T14:10:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:10:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:10:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:10:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap3 2023-12-04T14:10:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap3 2023-12-04T14:10:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:10:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap3 2023-12-04T14:10:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:10:21.263 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-04T14:10:21.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-04T14:10:21.264 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-04T14:10:21.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi176.stderr:2023-12-04T14:10:21.264+0000 7f39e3bd6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 91091) UID: 1000 2023-12-04T14:10:25.265 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-04T14:10:25.265 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:10:25.266 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-04T14:10:25.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-04T14:10:25.266 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-04T14:10:25.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-04T14:10:25.270 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-12-04T14:10:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-04T14:10:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-04T14:10:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:10:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:10:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:10:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:10:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-04T14:10:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:10:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:10:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:10:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-04T14:10:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:10:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:10:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:10:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:10:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:10:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:10:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:10:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:10:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:10:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:10:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:10:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:10:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:10:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:10:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:10:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:10:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:10:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:10:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:10:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:10:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:10:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:10:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:10:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:10:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:10:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:10:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:10:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:10:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:10:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:10:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:10:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:10:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:10:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:10:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:10:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:10:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:10:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:10:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:10:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:10:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:10:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:10:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:10:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:10:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:10:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:10:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:10:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:10:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:10:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:10:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:10:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:10:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:10:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:10:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:10:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:10:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:10:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:10:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:10:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:10:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:10:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:10:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:10:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:10:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:10:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:10:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:10:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:10:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:10:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:10:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:10:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-test.status 2023-12-04T14:10:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-test.status 2023-12-04T14:10:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:10:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:10:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.status 2023-12-04T14:10:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.status 2023-12-04T14:10:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:10:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:10:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-04T14:10:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:10:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-04T14:10:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 13 2023-12-04T14:10:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:10:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:10:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:10:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-04T14:10:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:10:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:10:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=13 2023-12-04T14:10:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:10:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:10:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:10:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:10:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:10:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:10:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:10:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:10:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:10:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:10:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:10:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:10:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:10:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 13 = '' 2023-12-04T14:10:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:10:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:10:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:10:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:10:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:10:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:10:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:10:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:10:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:10:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:10:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:10:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:10:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 13 = '' 2023-12-04T14:10:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:10:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:10:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:10:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:10:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:10:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:10:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:10:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:10:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:10:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:10:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:10:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 13 = '' 2023-12-04T14:10:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:10:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:10:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:10:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:10:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:10:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:10:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:10:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=13 2023-12-04T14:10:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 13 = 13 2023-12-04T14:10:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:10:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-12-04T14:10:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap3 2023-12-04T14:10:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:10:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:10:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:10:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-12-04T14:10:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'test': 2023-12-04T14:10:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 512 MiB in 128 objects 2023-12-04T14:10:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:10:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 6 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10ad877be9cc 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10ad877be9cc 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:09:46 2023 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:10:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:10:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:10:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:10:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snap2-clean ']' 2023-12-04T14:10:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-04T14:10:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap2-clean 2023-12-04T14:10:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:10:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:10:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:10:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:10:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:10:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:10:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:10:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:10:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:10:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:10:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:10:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap3 2023-12-04T14:10:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap3 2023-12-04T14:10:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:10:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:10:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:10:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:10:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:10:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:10:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:10:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:10:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:10:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:10:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:10:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ clean_snap_name=snap3-clean 2023-12-04T14:10:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-12-04T14:10:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:10:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap3-clean 2023-12-04T14:10:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-12-04T14:10:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:10:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror test 2023-12-04T14:10:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:10:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:10:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=9 2023-12-04T14:10:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:10:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-04T14:10:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 32596992 bytes 2023-12-04T14:10:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 80755712 bytes 2023-12-04T14:10:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 146786304 bytes 2023-12-04T14:10:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 224060416 bytes 2023-12-04T14:10:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 224670720 bytes 2023-12-04T14:10:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 279617536 bytes 2023-12-04T14:10:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 340070400 bytes 2023-12-04T14:10:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 340668416 bytes 2023-12-04T14:10:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 373902336 bytes 2023-12-04T14:10:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 433771520 bytes 2023-12-04T14:10:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 452173824 bytes 2023-12-04T14:10:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 524278784 bytes 2023-12-04T14:10:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 538277888 bytes 2023-12-04T14:10:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 562689024 bytes 2023-12-04T14:10:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 631716864 bytes 2023-12-04T14:10:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 736290816 bytes 2023-12-04T14:10:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 809822208 bytes 2023-12-04T14:10:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 846823424 bytes 2023-12-04T14:10:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 852401152 bytes 2023-12-04T14:10:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 854227968 bytes 2023-12-04T14:10:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 914287616 bytes 2023-12-04T14:10:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 957516800 bytes 2023-12-04T14:10:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 984003584 bytes 2023-12-04T14:10:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1043945472 bytes 2023-12-04T14:10:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1094077440 bytes 2023-12-04T14:10:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1132777472 bytes 2023-12-04T14:10:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1160472576 bytes 2023-12-04T14:10:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1189298176 bytes 2023-12-04T14:10:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1232994304 bytes 2023-12-04T14:10:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1279839232 bytes 2023-12-04T14:10:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1345275904 bytes 2023-12-04T14:10:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1410149376 bytes 2023-12-04T14:10:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1467600896 bytes 2023-12-04T14:10:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1565713408 bytes 2023-12-04T14:10:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1576546304 bytes 2023-12-04T14:10:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1646906368 bytes 2023-12-04T14:10:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1684658176 bytes 2023-12-04T14:10:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1693988864 bytes 2023-12-04T14:10:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1756307456 bytes 2023-12-04T14:10:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1805730816 bytes 2023-12-04T14:10:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1905694720 bytes 2023-12-04T14:10:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1966521344 bytes 2023-12-04T14:10:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1978396672 bytes 2023-12-04T14:10:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1999473664 bytes 2023-12-04T14:10:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 2015694848 bytes 2023-12-04T14:10:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 2016308224 bytes 2023-12-04T14:10:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2045689856 bytes 2023-12-04T14:10:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2068019200 bytes 2023-12-04T14:10:52.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2103818240 bytes 2023-12-04T14:10:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2181434368 bytes 2023-12-04T14:10:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2201690112 bytes 2023-12-04T14:10:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2220187648 bytes 2023-12-04T14:10:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2268875776 bytes 2023-12-04T14:10:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2301032448 bytes 2023-12-04T14:10:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2321753088 bytes 2023-12-04T14:10:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2399174656 bytes 2023-12-04T14:10:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2458123264 bytes 2023-12-04T14:10:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:10:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:10:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:10:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:10:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-04T14:10:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:10:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:10:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:10:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:10:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:10:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:10:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:10:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:10:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:10:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:10:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:10:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:10:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:10:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:10:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:10:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:10:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:10:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:10:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+replaying 2023-12-04T14:10:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":512023552.0,"last_snapshot_bytes":534590464,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701698980,"remote_snapshot_timestamp":1701698980,"replay_state":"idle"} 2023-12-04T14:10:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: service: mirror.2 on smithi176 2023-12-04T14:10:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:10:34 2023-12-04T14:10:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:10:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:10:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+stopped 2023-12-04T14:10:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: local image is primary 2023-12-04T14:10:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:10:34 2023-12-04T14:10:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:10:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:10:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ description: .*' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:10:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":512023552.0,"last_snapshot_bytes":534590464,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701698980,"remote_snapshot_timestamp":1701698980,"replay_state":"idle"} 2023-12-04T14:10:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:10:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:10:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^up+' 2023-12-04T14:10:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:10:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:10:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.2 on smithi176 2023-12-04T14:10:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:10:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'last_update=2023-12-04 14:10:34' 2023-12-04T14:10:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-04 14:10:34' 2023-12-04T14:10:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:10:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:10:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'prev_last_update=2023-12-04 14:10:34' 2023-12-04T14:10:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-04T14:10:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror.mirror_status 2023-12-04T14:10:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:10:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror.mirror_status 2023-12-04T14:10:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-04T14:10:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:10:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local last_update state 2023-12-04T14:10:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-04T14:10:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ last_update='2023-12-04 14:10:34' 2023-12-04T14:10:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-04T14:10:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=up+replaying 2023-12-04T14:10:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:10:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep up+replaying 2023-12-04T14:10:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:10:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:10:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:10:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap4 2023-12-04T14:10:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap4 2023-12-04T14:10:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:10:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:10:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:10:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap4 2023-12-04T14:10:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:11:03.276 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7532872014324595) > weight (0.25) 2023-12-04T14:11:03.277 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9009757505062845) > weight (0.25) 2023-12-04T14:11:03.277 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.885824439916652) > weight (0.25) 2023-12-04T14:11:03.277 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9028983378604438) > weight (0.25) 2023-12-04T14:11:03.277 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2023-12-04T14:11:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-04T14:11:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:11:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-04T14:11:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:11:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:11:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:11:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:11:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:11:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-04T14:11:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:11:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:11:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:11:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:11:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-04T14:11:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:11:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:11:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:11:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:11:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:11:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:11:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:11:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:11:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:11:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:11:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:11:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:11:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:11:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:11:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:11:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:11:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:11:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:11:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:11:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:11:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:11:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:11:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:11:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:11:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:11:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:11:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:11:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:11:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:11:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:11:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:11:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:11:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:11:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:11:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:11:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:11:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:11:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:11:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:11:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:11:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:11:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:11:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:11:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:11:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:11:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:11:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:11:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:11:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:11:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:11:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:11:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:11:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:11:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:11:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:11:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:11:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:11:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:11:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:11:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:11:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:11:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:11:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:11:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:11:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:11:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:11:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:11:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:11:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:11:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:11:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:11:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:11:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:11:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:11:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:11:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:11:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:11:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:11:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:11:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:11:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:11:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:11:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:11:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:11:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:11:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:11:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:11:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:11:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:11:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:11:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:11:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:11:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:11:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-04T14:11:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:11:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:11:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:11:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:11:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-04T14:11:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:11:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:11:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-test.status 2023-12-04T14:11:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-test.status 2023-12-04T14:11:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:11:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:11:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.status 2023-12-04T14:11:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.status 2023-12-04T14:11:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:11:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:11:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-04T14:11:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:11:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-04T14:11:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 17 2023-12-04T14:11:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:11:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:11:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:25.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:11:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-04T14:11:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:11:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:11:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=17 2023-12-04T14:11:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:11:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:11:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:11:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:11:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:11:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:11:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:11:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:11:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:11:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:11:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:11:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:11:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:11:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:11:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 17 = '' 2023-12-04T14:11:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:11:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:11:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:11:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:11:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:11:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:11:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:11:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:11:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:11:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:11:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:11:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:11:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:11:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 17 = '' 2023-12-04T14:11:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:11:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:11:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:11:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:11:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:11:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:11:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:11:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:11:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:11:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:11:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:11:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:11:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:11:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 17 = '' 2023-12-04T14:11:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:11:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:11:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:11:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:11:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:11:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:11:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:11:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:11:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=17 2023-12-04T14:11:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 17 = 17 2023-12-04T14:11:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:11:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-12-04T14:11:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:11:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap4 2023-12-04T14:11:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:11:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:11:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:11:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-12-04T14:11:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'test': 2023-12-04T14:11:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 512 MiB in 128 objects 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 8 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10ad877be9cc 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10ad877be9cc 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:09:46 2023 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:11:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:11:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:11:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:11:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:11:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snap3-clean ']' 2023-12-04T14:11:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-04T14:11:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap3-clean 2023-12-04T14:11:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:11:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:11:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:11:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:11:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:11:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:11:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:11:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:11:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:11:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:11:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:11:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap4 2023-12-04T14:11:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap4 2023-12-04T14:11:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:11:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:11:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:11:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:11:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:11:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:11:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:11:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:11:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:11:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:11:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:11:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ clean_snap_name=snap4-clean 2023-12-04T14:11:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-12-04T14:11:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:11:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap4-clean 2023-12-04T14:11:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-12-04T14:11:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:11:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror test 2023-12-04T14:11:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:11:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:11:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=8 2023-12-04T14:11:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:11:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-04T14:11:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 606208 bytes 2023-12-04T14:11:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 31752192 bytes 2023-12-04T14:11:33.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 70913024 bytes 2023-12-04T14:11:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 119386112 bytes 2023-12-04T14:11:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 196238336 bytes 2023-12-04T14:11:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 217909248 bytes 2023-12-04T14:11:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 233728000 bytes 2023-12-04T14:11:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 286429184 bytes 2023-12-04T14:11:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 335500288 bytes 2023-12-04T14:11:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 389239808 bytes 2023-12-04T14:11:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 407717888 bytes 2023-12-04T14:11:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 436598784 bytes 2023-12-04T14:11:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 453045248 bytes 2023-12-04T14:11:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 484835328 bytes 2023-12-04T14:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 521993216 bytes 2023-12-04T14:11:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 522606592 bytes 2023-12-04T14:11:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 611792896 bytes 2023-12-04T14:11:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 623677440 bytes 2023-12-04T14:11:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 719839232 bytes 2023-12-04T14:11:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 749019136 bytes 2023-12-04T14:11:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 754184192 bytes 2023-12-04T14:11:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 764209152 bytes 2023-12-04T14:11:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 872245248 bytes 2023-12-04T14:11:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 916316160 bytes 2023-12-04T14:11:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 950276096 bytes 2023-12-04T14:11:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 988527616 bytes 2023-12-04T14:11:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1018301440 bytes 2023-12-04T14:11:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1057756160 bytes 2023-12-04T14:11:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1086590976 bytes 2023-12-04T14:11:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1172389888 bytes 2023-12-04T14:11:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1244630016 bytes 2023-12-04T14:11:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1322954752 bytes 2023-12-04T14:11:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1390432256 bytes 2023-12-04T14:11:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1421925376 bytes 2023-12-04T14:11:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1487247360 bytes 2023-12-04T14:11:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1502633984 bytes 2023-12-04T14:11:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1562205184 bytes 2023-12-04T14:11:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1603035136 bytes 2023-12-04T14:11:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1664049152 bytes 2023-12-04T14:11:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1722432512 bytes 2023-12-04T14:11:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1776928768 bytes 2023-12-04T14:11:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1831308288 bytes 2023-12-04T14:11:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1868906496 bytes 2023-12-04T14:11:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1881345024 bytes 2023-12-04T14:11:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1925385216 bytes 2023-12-04T14:11:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1933348864 bytes 2023-12-04T14:11:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1992348672 bytes 2023-12-04T14:11:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2049437696 bytes 2023-12-04T14:11:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2056138752 bytes 2023-12-04T14:11:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2099350528 bytes 2023-12-04T14:11:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:11:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:11:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:11:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:11:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-04T14:11:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:11:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:11:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:11:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:11:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:11:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:11:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:11:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:11:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:11:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:11:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:11:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:11:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:11:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:11:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:11:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:11:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:11:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:11:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:11:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:11:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+replaying 2023-12-04T14:11:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: replaying, {"bytes_per_second":17829819.73,"bytes_per_snapshot":534037504.0,"last_snapshot_bytes":534894592,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701699084,"remote_snapshot_timestamp":1701699084,"replay_state":"idle"} 2023-12-04T14:11:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: service: mirror.2 on smithi176 2023-12-04T14:11:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:11:34 2023-12-04T14:11:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:11:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:11:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+stopped 2023-12-04T14:11:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: local image is primary 2023-12-04T14:11:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:11:34 2023-12-04T14:11:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:11:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:11:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ description: .*' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:11:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":17829819.73,"bytes_per_snapshot":534037504.0,"last_snapshot_bytes":534894592,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701699084,"remote_snapshot_timestamp":1701699084,"replay_state":"idle"} 2023-12-04T14:11:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:11:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:11:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^up+' 2023-12-04T14:11:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:11:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:11:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.2 on smithi176 2023-12-04T14:11:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:11:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'last_update=2023-12-04 14:11:34' 2023-12-04T14:11:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-04 14:11:34' 2023-12-04T14:11:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:11:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:11:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'prev_last_update=2023-12-04 14:11:34' 2023-12-04T14:11:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-04T14:11:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror.mirror_status 2023-12-04T14:11:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:11:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror.mirror_status 2023-12-04T14:11:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-04T14:11:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:11:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local last_update state 2023-12-04T14:11:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-04T14:11:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ last_update='2023-12-04 14:11:34' 2023-12-04T14:11:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-04T14:11:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=up+replaying 2023-12-04T14:11:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:11:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep up+replaying 2023-12-04T14:11:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:11:42.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:11:42.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:11:42.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap5 2023-12-04T14:11:42.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap5 2023-12-04T14:11:42.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:11:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:11:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:11:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap5 2023-12-04T14:11:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:12:06.278 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-04T14:12:06.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-04T14:12:06.279 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-04T14:12:06.279 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-04T14:12:06.279 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:12:06.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi176.stderr:2023-12-04T14:12:06.279+0000 7fec25368640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 91023) UID: 1000 2023-12-04T14:12:07.271 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7067797313425951) > weight (0.25) 2023-12-04T14:12:07.271 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6756116433904866) > weight (0.25) 2023-12-04T14:12:07.271 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.27362386673949257) > weight (0.25) 2023-12-04T14:12:07.271 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8489583683387583) > weight (0.25) 2023-12-04T14:12:07.271 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2023-12-04T14:12:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-04T14:12:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-04T14:12:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:12:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:12:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:12:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-04T14:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-04T14:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:12:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:12:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:12:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:12:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:12:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:12:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:12:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:12:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:12:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:12:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:12:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:12:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:12:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:12:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:12:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:12:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:12:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:12:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:12:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:12:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:12:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:12:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:12:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:12:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:12:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:12:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:12:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:12:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:12:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:12:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-04T14:12:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:12:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:12:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:12:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:12:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:12:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:12:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:12:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:12:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:12:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:12:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:12:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:12:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:12:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:12:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:12:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:12:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:12:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:12:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:12:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:12:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:12:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:12:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:12:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:12:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:12:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:12:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:12:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:12:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:12:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:12:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:12:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:12:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:12:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:12:10.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:12:10.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:12:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:12:10.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:12:10.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:12:10.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:12:10.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-04T14:12:10.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-test.status 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-test.status 2023-12-04T14:12:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:12:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:12:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.status 2023-12-04T14:12:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.status 2023-12-04T14:12:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:12:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-04T14:12:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:12:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-04T14:12:12.382 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-04T14:12:12.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-04T14:12:12.382 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-04T14:12:12.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-04T14:12:12.386 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2023-12-04T14:12:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 21 2023-12-04T14:12:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:12:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:12:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:12:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-04T14:12:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:12:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:12:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=21 2023-12-04T14:12:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:12:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:12:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:12:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:12:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:12:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:12:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:12:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:12:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:12:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:12:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 21 = '' 2023-12-04T14:12:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:12:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:12:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:12:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:12:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:12:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:12:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:12:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:12:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:12:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:12:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 21 = '' 2023-12-04T14:12:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:12:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:12:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:12:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:12:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:12:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:12:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:12:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:12:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:12:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:12:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 21 = '' 2023-12-04T14:12:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:12:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:12:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:12:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:12:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:12:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=21 2023-12-04T14:12:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 21 = 21 2023-12-04T14:12:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:12:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-12-04T14:12:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap5 2023-12-04T14:12:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:12:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:12:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:12:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-12-04T14:12:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'test': 2023-12-04T14:12:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 512 MiB in 128 objects 2023-12-04T14:12:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:12:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 10 2023-12-04T14:12:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10ad877be9cc 2023-12-04T14:12:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10ad877be9cc 2023-12-04T14:12:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:12:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:12:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:12:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:12:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:12:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:11:30 2023 2023-12-04T14:12:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:12:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:12:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:12:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:12:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:12:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:12:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:12:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snap4-clean ']' 2023-12-04T14:12:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-04T14:12:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap4-clean 2023-12-04T14:12:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:12:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:12:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:12:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:12:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:12:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:12:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:12:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:12:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:12:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:12:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:12:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap5 2023-12-04T14:12:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap5 2023-12-04T14:12:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:12:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:12:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:12:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:12:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:12:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:12:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:12:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:12:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:12:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:12:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:12:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ clean_snap_name=snap5-clean 2023-12-04T14:12:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-12-04T14:12:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:12:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap5-clean 2023-12-04T14:12:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-12-04T14:12:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:12:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror test 2023-12-04T14:12:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:12:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:12:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=5 2023-12-04T14:12:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:12:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-04T14:12:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 76929024 bytes 2023-12-04T14:12:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 129471488 bytes 2023-12-04T14:12:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 181364736 bytes 2023-12-04T14:12:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 257698816 bytes 2023-12-04T14:12:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 328687616 bytes 2023-12-04T14:12:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 429488128 bytes 2023-12-04T14:12:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 494726144 bytes 2023-12-04T14:12:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 521612288 bytes 2023-12-04T14:12:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 554930176 bytes 2023-12-04T14:12:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 590799872 bytes 2023-12-04T14:12:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 637498368 bytes 2023-12-04T14:12:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 651593728 bytes 2023-12-04T14:12:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 719246336 bytes 2023-12-04T14:12:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 788207616 bytes 2023-12-04T14:12:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 829179904 bytes 2023-12-04T14:12:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 848248832 bytes 2023-12-04T14:12:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 878663680 bytes 2023-12-04T14:12:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 915042304 bytes 2023-12-04T14:12:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 929713152 bytes 2023-12-04T14:12:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 966307840 bytes 2023-12-04T14:12:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 1005063168 bytes 2023-12-04T14:12:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1021484032 bytes 2023-12-04T14:12:24.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1038982144 bytes 2023-12-04T14:12:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1090399232 bytes 2023-12-04T14:12:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1171693568 bytes 2023-12-04T14:12:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1240860672 bytes 2023-12-04T14:12:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1315504128 bytes 2023-12-04T14:12:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1381579776 bytes 2023-12-04T14:12:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:12:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:12:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:12:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:12:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-04T14:12:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:12:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:12:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:12:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:12:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:12:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:12:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:12:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:12:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:12:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:12:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:12:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:12:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:12:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:12:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:12:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:12:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:12:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:12:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+replaying 2023-12-04T14:12:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: replaying, {"bytes_per_second":17829819.73,"bytes_per_snapshot":534037504.0,"last_snapshot_bytes":534894592,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701699084,"remote_snapshot_timestamp":1701699084,"replay_state":"idle"} 2023-12-04T14:12:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: service: mirror.2 on smithi176 2023-12-04T14:12:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:12:04 2023-12-04T14:12:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:12:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:12:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+stopped 2023-12-04T14:12:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: local image is primary 2023-12-04T14:12:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:12:04 2023-12-04T14:12:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:12:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:12:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ description: .*' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:12:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":17829819.73,"bytes_per_snapshot":534037504.0,"last_snapshot_bytes":534894592,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701699084,"remote_snapshot_timestamp":1701699084,"replay_state":"idle"} 2023-12-04T14:12:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:12:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:12:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^up+' 2023-12-04T14:12:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:12:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:12:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.2 on smithi176 2023-12-04T14:12:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:12:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'last_update=2023-12-04 14:12:04' 2023-12-04T14:12:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-04 14:12:04' 2023-12-04T14:12:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:12:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'prev_last_update=2023-12-04 14:12:04' 2023-12-04T14:12:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-04T14:12:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror.mirror_status 2023-12-04T14:12:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:12:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror.mirror_status 2023-12-04T14:12:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-04T14:12:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:12:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local last_update state 2023-12-04T14:12:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-04T14:12:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ last_update='2023-12-04 14:12:04' 2023-12-04T14:12:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-04T14:12:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=up+replaying 2023-12-04T14:12:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:12:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep up+replaying 2023-12-04T14:12:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:12:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:12:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:12:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap6 2023-12-04T14:12:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap6 2023-12-04T14:12:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:12:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap6 2023-12-04T14:12:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:12:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-04T14:12:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-04T14:12:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:12:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:12:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:12:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:12:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-04T14:12:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:12:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:12:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:12:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-04T14:12:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:12:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:12:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:12:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:12:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:12:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:12:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:12:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:12:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:12:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:12:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:12:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:12:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:12:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:12:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:12:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:12:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:12:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:12:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:12:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:12:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:12:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:12:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:12:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:12:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:12:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:12:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:12:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:12:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:12:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:12:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:12:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:12:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:12:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:12:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:12:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:12:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:12:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:12:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:12:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:12:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:12:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:12:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:12:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:12:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:12:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:12:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:12:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:12:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:12:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:12:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:12:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:12:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:12:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:12:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:12:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:12:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:12:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:12:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:12:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:12:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:12:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:12:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:12:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:12:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:12:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:12:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:12:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:12:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:12:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:12:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:12:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:12:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:12:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:12:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:12:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:12:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:12:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:12:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:12:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:12:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:12:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:12:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:12:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:12:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:12:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:12:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:12:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:12:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:12:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:12:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:12:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:12:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:12:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:12:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:12:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:12:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:12:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:12:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:12:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:12:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:12:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:12:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:12:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:12:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:12:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:12:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-04T14:12:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-test.status 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-test.status 2023-12-04T14:12:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:12:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:12:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.status 2023-12-04T14:12:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.status 2023-12-04T14:12:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:12:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-04T14:12:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:12:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-04T14:12:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 25 2023-12-04T14:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-04T14:12:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:12:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:12:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=25 2023-12-04T14:12:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:12:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:12:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:12:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:12:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:12:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:12:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:12:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:12:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:12:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:12:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:12:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 25 = '' 2023-12-04T14:12:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:12:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:12:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:12:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:12:55.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:12:55.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:12:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:12:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:12:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:12:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:12:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 25 = '' 2023-12-04T14:12:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:12:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:12:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:12:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:12:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:12:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:12:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:12:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:12:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:12:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:12:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 25 = '' 2023-12-04T14:12:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:12:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:12:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:12:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:12:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:12:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:12:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=25 2023-12-04T14:12:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 25 = 25 2023-12-04T14:12:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:12:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-12-04T14:12:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:12:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap6 2023-12-04T14:12:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:12:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:12:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:12:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-12-04T14:12:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'test': 2023-12-04T14:12:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 512 MiB in 128 objects 2023-12-04T14:12:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:12:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 12 2023-12-04T14:12:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10ad877be9cc 2023-12-04T14:12:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10ad877be9cc 2023-12-04T14:12:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:12:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:12:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:12:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:12:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:12:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:11:30 2023 2023-12-04T14:12:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:12:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:12:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:12:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:12:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:12:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:12:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:12:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snap5-clean ']' 2023-12-04T14:12:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-04T14:12:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:12:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:12:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap5-clean 2023-12-04T14:12:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:12:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:12:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:12:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:12:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:12:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:12:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:13:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:13:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap6 2023-12-04T14:13:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap6 2023-12-04T14:13:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:13:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:13:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:13:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:13:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:13:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:13:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ clean_snap_name=snap6-clean 2023-12-04T14:13:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-12-04T14:13:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:13:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap6-clean 2023-12-04T14:13:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-12-04T14:13:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:13:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror test 2023-12-04T14:13:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:13:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:13:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=13 2023-12-04T14:13:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:13:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-04T14:13:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 66461696 bytes 2023-12-04T14:13:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 68554752 bytes 2023-12-04T14:13:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 88147968 bytes 2023-12-04T14:13:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 102253568 bytes 2023-12-04T14:13:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 173547520 bytes 2023-12-04T14:13:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 221590528 bytes 2023-12-04T14:13:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 244547584 bytes 2023-12-04T14:13:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 256679936 bytes 2023-12-04T14:13:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 322786304 bytes 2023-12-04T14:13:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 364852224 bytes 2023-12-04T14:13:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 389425152 bytes 2023-12-04T14:13:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 434428928 bytes 2023-12-04T14:13:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 435052544 bytes 2023-12-04T14:13:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 453617664 bytes 2023-12-04T14:13:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 454253568 bytes 2023-12-04T14:13:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 563461120 bytes 2023-12-04T14:13:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 564067328 bytes 2023-12-04T14:13:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 608509952 bytes 2023-12-04T14:13:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 638316544 bytes 2023-12-04T14:13:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 679723008 bytes 2023-12-04T14:13:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 687065088 bytes 2023-12-04T14:13:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 748661760 bytes 2023-12-04T14:13:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 789294080 bytes 2023-12-04T14:13:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 831564800 bytes 2023-12-04T14:13:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 873279488 bytes 2023-12-04T14:13:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 903748608 bytes 2023-12-04T14:13:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 932945920 bytes 2023-12-04T14:13:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 955245568 bytes 2023-12-04T14:13:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1064948736 bytes 2023-12-04T14:13:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1088402432 bytes 2023-12-04T14:13:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1168555008 bytes 2023-12-04T14:13:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1199087616 bytes 2023-12-04T14:13:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1229225984 bytes 2023-12-04T14:13:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1236124672 bytes 2023-12-04T14:13:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1302463488 bytes 2023-12-04T14:13:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1368177664 bytes 2023-12-04T14:13:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1388697600 bytes 2023-12-04T14:13:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1469999104 bytes 2023-12-04T14:13:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1470611456 bytes 2023-12-04T14:13:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1496212480 bytes 2023-12-04T14:13:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1543866368 bytes 2023-12-04T14:13:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1603175424 bytes 2023-12-04T14:13:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1674704896 bytes 2023-12-04T14:13:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1706501120 bytes 2023-12-04T14:13:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1741165568 bytes 2023-12-04T14:13:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1774887936 bytes 2023-12-04T14:13:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1862568960 bytes 2023-12-04T14:13:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1884665856 bytes 2023-12-04T14:13:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 1909567488 bytes 2023-12-04T14:13:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 1976201216 bytes 2023-12-04T14:13:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2002316288 bytes 2023-12-04T14:13:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2055671808 bytes 2023-12-04T14:13:10.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2092242944 bytes 2023-12-04T14:13:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2150497280 bytes 2023-12-04T14:13:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2175330304 bytes 2023-12-04T14:13:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2215724032 bytes 2023-12-04T14:13:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2218500096 bytes 2023-12-04T14:13:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2263333888 bytes 2023-12-04T14:13:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2286806016 bytes 2023-12-04T14:13:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2372387840 bytes 2023-12-04T14:13:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2416527360 bytes 2023-12-04T14:13:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2453475328 bytes 2023-12-04T14:13:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2495728640 bytes 2023-12-04T14:13:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2510043136 bytes 2023-12-04T14:13:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2545848320 bytes 2023-12-04T14:13:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2597449728 bytes 2023-12-04T14:13:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2618155008 bytes 2023-12-04T14:13:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2657501184 bytes 2023-12-04T14:13:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2671627264 bytes 2023-12-04T14:13:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 2710975488 bytes 2023-12-04T14:13:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 2737968128 bytes 2023-12-04T14:13:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 2738577408 bytes 2023-12-04T14:13:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7300 IOs, 2815654912 bytes 2023-12-04T14:13:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7400 IOs, 2862256128 bytes 2023-12-04T14:13:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7500 IOs, 2911498240 bytes 2023-12-04T14:13:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7600 IOs, 2958830592 bytes 2023-12-04T14:13:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7700 IOs, 2975683584 bytes 2023-12-04T14:13:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7800 IOs, 2999429120 bytes 2023-12-04T14:13:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7900 IOs, 3098706944 bytes 2023-12-04T14:13:14.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8000 IOs, 3128576000 bytes 2023-12-04T14:13:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8100 IOs, 3188820992 bytes 2023-12-04T14:13:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8200 IOs, 3251815424 bytes 2023-12-04T14:13:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8300 IOs, 3277800448 bytes 2023-12-04T14:13:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:13:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:13:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:13:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:13:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-04T14:13:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:13:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:13:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:13:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:13:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:13:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:13:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:13:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:13:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:13:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:13:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:13:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:13:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:13:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:13:16.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:13:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:13:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:13:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:13:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:13:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:13:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+replaying 2023-12-04T14:13:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: replaying, {"bytes_per_second":16437623.47,"bytes_per_snapshot":514810368.0,"last_snapshot_bytes":493128704,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701699173,"remote_snapshot_timestamp":1701699173,"replay_state":"idle"} 2023-12-04T14:13:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: service: mirror.2 on smithi176 2023-12-04T14:13:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:13:04 2023-12-04T14:13:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:13:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:13:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+stopped 2023-12-04T14:13:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: local image is primary 2023-12-04T14:13:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:13:04 2023-12-04T14:13:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:13:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:13:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ description: .*' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:13:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":16437623.47,"bytes_per_snapshot":514810368.0,"last_snapshot_bytes":493128704,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701699173,"remote_snapshot_timestamp":1701699173,"replay_state":"idle"} 2023-12-04T14:13:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:13:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:13:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^up+' 2023-12-04T14:13:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:13:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:13:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.2 on smithi176 2023-12-04T14:13:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:13:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'last_update=2023-12-04 14:13:04' 2023-12-04T14:13:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-04 14:13:04' 2023-12-04T14:13:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:13:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:13:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'prev_last_update=2023-12-04 14:13:04' 2023-12-04T14:13:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-04T14:13:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror.mirror_status 2023-12-04T14:13:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:13:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror.mirror_status 2023-12-04T14:13:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-04T14:13:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:13:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local last_update state 2023-12-04T14:13:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-04T14:13:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ last_update='2023-12-04 14:13:04' 2023-12-04T14:13:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-04T14:13:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=up+replaying 2023-12-04T14:13:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:13:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep up+replaying 2023-12-04T14:13:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:13:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:13:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:13:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap7 2023-12-04T14:13:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap7 2023-12-04T14:13:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:13:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap7 2023-12-04T14:13:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:13:23.387 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8064786325399472) > weight (0.25) 2023-12-04T14:13:23.387 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-04T14:13:23.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-04T14:13:23.388 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-04T14:13:23.388 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-04T14:13:23.388 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:13:23.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi176.stderr:2023-12-04T14:13:23.387+0000 7fb944e4a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 93326) UID: 1000 2023-12-04T14:13:29.490 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-04T14:13:29.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-04T14:13:29.490 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-04T14:13:29.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-04T14:13:29.493 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-04T14:13:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-04T14:13:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:13:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-04T14:13:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:13:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:13:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:13:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:13:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:13:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-04T14:13:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:13:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:13:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:13:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:13:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-04T14:13:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:13:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:13:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:13:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:13:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:13:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:13:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:13:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:13:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:13:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:13:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:13:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:13:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:13:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:13:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:13:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:13:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:13:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:13:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:13:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:13:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:13:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:13:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:13:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:13:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:13:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:13:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:13:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:13:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:13:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:13:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:13:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:13:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:13:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:13:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:13:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:13:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:13:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:13:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:13:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:13:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:13:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:13:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:13:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:13:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:13:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:13:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:13:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:13:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:13:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:13:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:13:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:13:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:13:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:13:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:13:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:13:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:13:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:13:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:13:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:13:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:13:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:13:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:13:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:13:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:13:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:13:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-04T14:13:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:13:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:13:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:13:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:13:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-04T14:13:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:13:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:13:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-test.status 2023-12-04T14:13:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-test.status 2023-12-04T14:13:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:13:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:13:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.status 2023-12-04T14:13:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.status 2023-12-04T14:13:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:13:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:13:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-04T14:13:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:13:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-04T14:13:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 29 2023-12-04T14:13:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:13:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:13:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:13:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-04T14:13:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:13:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:13:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=29 2023-12-04T14:13:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:13:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:13:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:13:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:13:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:13:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:13:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:13:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:13:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:13:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:13:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:13:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:13:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:13:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:13:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 29 = '' 2023-12-04T14:13:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:13:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:13:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:13:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:13:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:13:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:13:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:13:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:13:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:13:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:13:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:13:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:13:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:13:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 29 = '' 2023-12-04T14:13:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:13:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:13:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:13:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:13:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:13:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:13:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:13:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:13:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:13:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 29 = '' 2023-12-04T14:13:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:13:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:13:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:13:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:13:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:13:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:13:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:13:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:13:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=29 2023-12-04T14:13:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 29 = 29 2023-12-04T14:13:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:13:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-12-04T14:13:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:13:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap7 2023-12-04T14:13:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:13:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:13:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:13:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-12-04T14:13:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'test': 2023-12-04T14:13:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 512 MiB in 128 objects 2023-12-04T14:13:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:13:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 14 2023-12-04T14:13:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10ad877be9cc 2023-12-04T14:13:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10ad877be9cc 2023-12-04T14:13:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:13:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:13:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:13:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:13:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:13:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:12:59 2023 2023-12-04T14:13:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:13:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:13:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:13:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:13:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:13:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:13:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:13:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snap6-clean ']' 2023-12-04T14:13:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-04T14:13:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap6-clean 2023-12-04T14:13:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:13:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:13:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:13:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:13:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:13:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:13:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap7 2023-12-04T14:13:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap7 2023-12-04T14:13:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:13:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:13:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:13:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:13:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:13:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:13:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:13:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ clean_snap_name=snap7-clean 2023-12-04T14:13:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-12-04T14:13:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:13:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap7-clean 2023-12-04T14:13:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-12-04T14:13:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:13:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror test 2023-12-04T14:13:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:13:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:13:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:13:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:13:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=13 2023-12-04T14:13:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:13:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-04T14:13:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 2216960 bytes 2023-12-04T14:13:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 43991040 bytes 2023-12-04T14:13:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 55980032 bytes 2023-12-04T14:13:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 75004928 bytes 2023-12-04T14:13:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 121036800 bytes 2023-12-04T14:13:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 162187264 bytes 2023-12-04T14:13:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 213686272 bytes 2023-12-04T14:13:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 288515072 bytes 2023-12-04T14:13:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 289135616 bytes 2023-12-04T14:13:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 299643904 bytes 2023-12-04T14:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 313602048 bytes 2023-12-04T14:13:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 339522560 bytes 2023-12-04T14:13:59.272 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5594933513815737) > weight (0.25) 2023-12-04T14:13:59.272 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-04T14:13:59.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-04T14:13:59.272 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-04T14:13:59.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr:2023-12-04T14:13:59.271+0000 7f8a1615f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 91109) UID: 1000 2023-12-04T14:13:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 380989440 bytes 2023-12-04T14:13:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 464171008 bytes 2023-12-04T14:13:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 499374080 bytes 2023-12-04T14:13:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 525889536 bytes 2023-12-04T14:14:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 569231360 bytes 2023-12-04T14:14:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 602092544 bytes 2023-12-04T14:14:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 644052992 bytes 2023-12-04T14:14:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 685206528 bytes 2023-12-04T14:14:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 685807616 bytes 2023-12-04T14:14:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 755301376 bytes 2023-12-04T14:14:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 793113600 bytes 2023-12-04T14:14:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 846076928 bytes 2023-12-04T14:14:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 963556352 bytes 2023-12-04T14:14:01.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 988497920 bytes 2023-12-04T14:14:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1116441600 bytes 2023-12-04T14:14:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1188635648 bytes 2023-12-04T14:14:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1224558592 bytes 2023-12-04T14:14:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1299827712 bytes 2023-12-04T14:14:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1306806272 bytes 2023-12-04T14:14:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1361123328 bytes 2023-12-04T14:14:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1384813568 bytes 2023-12-04T14:14:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1423248384 bytes 2023-12-04T14:14:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1478270976 bytes 2023-12-04T14:14:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1478902784 bytes 2023-12-04T14:14:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1487236096 bytes 2023-12-04T14:14:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1604799488 bytes 2023-12-04T14:14:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1674502144 bytes 2023-12-04T14:14:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1700609024 bytes 2023-12-04T14:14:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1728633856 bytes 2023-12-04T14:14:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1790388224 bytes 2023-12-04T14:14:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1817807872 bytes 2023-12-04T14:14:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1862061056 bytes 2023-12-04T14:14:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1899915264 bytes 2023-12-04T14:14:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1922695168 bytes 2023-12-04T14:14:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1973720064 bytes 2023-12-04T14:14:05.274 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-04T14:14:05.274 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:14:05.274 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-04T14:14:05.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-04T14:14:05.275 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-04T14:14:05.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-04T14:14:05.278 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-12-04T14:14:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2051116032 bytes 2023-12-04T14:14:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2100179968 bytes 2023-12-04T14:14:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2122243072 bytes 2023-12-04T14:14:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2190456832 bytes 2023-12-04T14:14:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2207212544 bytes 2023-12-04T14:14:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2251993088 bytes 2023-12-04T14:14:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2297717760 bytes 2023-12-04T14:14:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2367854592 bytes 2023-12-04T14:14:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2420893696 bytes 2023-12-04T14:14:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2455208960 bytes 2023-12-04T14:14:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2513638400 bytes 2023-12-04T14:14:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2542728192 bytes 2023-12-04T14:14:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2554527744 bytes 2023-12-04T14:14:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2614290432 bytes 2023-12-04T14:14:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2655625216 bytes 2023-12-04T14:14:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2671137792 bytes 2023-12-04T14:14:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2808377344 bytes 2023-12-04T14:14:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2875896832 bytes 2023-12-04T14:14:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2907553792 bytes 2023-12-04T14:14:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2972075008 bytes 2023-12-04T14:14:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2997816320 bytes 2023-12-04T14:14:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 3113435136 bytes 2023-12-04T14:14:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 3163941888 bytes 2023-12-04T14:14:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 3202900992 bytes 2023-12-04T14:14:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 3224899584 bytes 2023-12-04T14:14:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7300 IOs, 3234936832 bytes 2023-12-04T14:14:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7400 IOs, 3254587392 bytes 2023-12-04T14:14:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7500 IOs, 3289453568 bytes 2023-12-04T14:14:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7600 IOs, 3349380096 bytes 2023-12-04T14:14:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7700 IOs, 3369482240 bytes 2023-12-04T14:14:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7800 IOs, 3388017664 bytes 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:14:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:14:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:14:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:14:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:14:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:14:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:14:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:14:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:14:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:14:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:14:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:14:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:14:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:14:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:14:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+replaying 2023-12-04T14:14:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: replaying, {"bytes_per_second":17790054.4,"bytes_per_snapshot":513415168.0,"last_snapshot_bytes":533701632,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701699227,"remote_snapshot_timestamp":1701699227,"replay_state":"idle"} 2023-12-04T14:14:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: service: mirror.2 on smithi176 2023-12-04T14:14:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:14:04 2023-12-04T14:14:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:14:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:14:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+stopped 2023-12-04T14:14:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: local image is primary 2023-12-04T14:14:11.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:14:04 2023-12-04T14:14:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:14:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:14:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ description: .*' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:14:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":17790054.4,"bytes_per_snapshot":513415168.0,"last_snapshot_bytes":533701632,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701699227,"remote_snapshot_timestamp":1701699227,"replay_state":"idle"} 2023-12-04T14:14:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:14:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:14:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^up+' 2023-12-04T14:14:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:14:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:14:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.2 on smithi176 2023-12-04T14:14:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:14:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'last_update=2023-12-04 14:14:04' 2023-12-04T14:14:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-04 14:14:04' 2023-12-04T14:14:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:14:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'prev_last_update=2023-12-04 14:14:04' 2023-12-04T14:14:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-04T14:14:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror.mirror_status 2023-12-04T14:14:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:14:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror.mirror_status 2023-12-04T14:14:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-04T14:14:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:14:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local last_update state 2023-12-04T14:14:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-04T14:14:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ last_update='2023-12-04 14:14:04' 2023-12-04T14:14:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-04T14:14:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=up+replaying 2023-12-04T14:14:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:14:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep up+replaying 2023-12-04T14:14:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:14:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:14:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:14:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap8 2023-12-04T14:14:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap8 2023-12-04T14:14:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:14:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap8 2023-12-04T14:14:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:14:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-04T14:14:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-04T14:14:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:14:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:14:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:14:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:14:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-04T14:14:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:14:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:14:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:14:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-04T14:14:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:14:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:14:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:14:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:14:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:14:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:14:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:14:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:14:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:14:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:14:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:14:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:14:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:14:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:14:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:14:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:14:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:14:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:14:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:14:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:14:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:14:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:14:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:14:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:14:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:14:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:14:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:14:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:14:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:14:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:14:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:14:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:14:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:14:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:14:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:14:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:14:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:14:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:14:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:14:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:14:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:14:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:14:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:14:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:14:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:14:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:14:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:14:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:14:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:14:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:14:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:14:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:14:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:14:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:14:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:14:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:14:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:14:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:14:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:14:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:14:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-04T14:14:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:14:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:14:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:14:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:14:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:14:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:14:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:14:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:14:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:14:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:14:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:14:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:14:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:14:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:14:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:14:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:14:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:14:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:14:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:14:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-04T14:14:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:14:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:14:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:14:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:14:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:14:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:14:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-test.status 2023-12-04T14:14:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-test.status 2023-12-04T14:14:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:14:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:14:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.status 2023-12-04T14:14:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.status 2023-12-04T14:14:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:14:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-04T14:14:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:14:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-04T14:14:42.494 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7097194892478689) > weight (0.25) 2023-12-04T14:14:42.495 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3229255276624543) > weight (0.25) 2023-12-04T14:14:42.495 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-04T14:14:42.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-04T14:14:42.495 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-04T14:14:42.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr:2023-12-04T14:14:42.494+0000 7f1219f1f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 91056) UID: 1000 2023-12-04T14:14:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 33 2023-12-04T14:14:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:14:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:14:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:14:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-04T14:14:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:14:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=33 2023-12-04T14:14:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:14:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:14:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:14:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:14:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:14:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:14:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=29 2023-12-04T14:14:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 33 = 29 2023-12-04T14:14:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:14:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:14:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:14:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:14:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:14:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=29 2023-12-04T14:14:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 33 = 29 2023-12-04T14:14:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:14:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:14:44.496 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-04T14:14:44.496 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:14:44.496 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-04T14:14:44.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-04T14:14:44.496 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-04T14:14:44.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-04T14:14:44.499 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2023-12-04T14:14:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:14:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:14:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:14:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=29 2023-12-04T14:14:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 33 = 29 2023-12-04T14:14:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:14:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:14:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:14:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:14:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:14:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=29 2023-12-04T14:14:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 33 = 29 2023-12-04T14:14:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:14:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 2 2023-12-04T14:14:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:14:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:14:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:14:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:14:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:14:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:14:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:14:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:14:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 33 = '' 2023-12-04T14:14:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:14:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 2 2023-12-04T14:14:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:14:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:14:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:14:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:14:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=33 2023-12-04T14:14:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 33 = 33 2023-12-04T14:14:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:14:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-12-04T14:14:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:14:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap8 2023-12-04T14:14:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:14:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:14:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:14:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-12-04T14:14:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'test': 2023-12-04T14:14:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 512 MiB in 128 objects 2023-12-04T14:14:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:14:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 16 2023-12-04T14:14:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10ad877be9cc 2023-12-04T14:14:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10ad877be9cc 2023-12-04T14:14:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:14:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:14:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:14:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:14:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:14:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:12:59 2023 2023-12-04T14:14:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:14:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:14:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:14:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:14:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:14:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:14:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:14:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snap7-clean ']' 2023-12-04T14:14:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-04T14:14:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap7-clean 2023-12-04T14:14:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:14:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:14:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:14:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:14:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:14:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:14:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:14:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:14:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:14:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:14:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:14:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap8 2023-12-04T14:14:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap8 2023-12-04T14:14:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:14:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:14:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:14:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:14:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:14:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:14:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:14:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:14:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:14:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:14:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:14:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ clean_snap_name=snap8-clean 2023-12-04T14:14:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-12-04T14:14:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:14:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap8-clean 2023-12-04T14:14:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-12-04T14:14:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:14:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror test 2023-12-04T14:14:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:14:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:14:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:14:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:14:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=8 2023-12-04T14:14:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:14:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-04T14:14:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 22368256 bytes 2023-12-04T14:14:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 45060096 bytes 2023-12-04T14:14:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 62913536 bytes 2023-12-04T14:14:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 92478464 bytes 2023-12-04T14:14:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 93094912 bytes 2023-12-04T14:14:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 165841920 bytes 2023-12-04T14:14:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 205922304 bytes 2023-12-04T14:14:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 226504704 bytes 2023-12-04T14:14:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 250955776 bytes 2023-12-04T14:14:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 288060416 bytes 2023-12-04T14:14:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 329072640 bytes 2023-12-04T14:14:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 390445056 bytes 2023-12-04T14:14:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 445748224 bytes 2023-12-04T14:14:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 491660288 bytes 2023-12-04T14:14:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 526040064 bytes 2023-12-04T14:14:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 602539008 bytes 2023-12-04T14:14:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 603155456 bytes 2023-12-04T14:14:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 633563136 bytes 2023-12-04T14:14:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 723069952 bytes 2023-12-04T14:14:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 741608448 bytes 2023-12-04T14:14:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 833093632 bytes 2023-12-04T14:14:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 846095360 bytes 2023-12-04T14:14:59.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 905877504 bytes 2023-12-04T14:14:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 922033152 bytes 2023-12-04T14:14:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 985635840 bytes 2023-12-04T14:14:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1001962496 bytes 2023-12-04T14:15:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1029985280 bytes 2023-12-04T14:15:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1085902848 bytes 2023-12-04T14:15:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1102771200 bytes 2023-12-04T14:15:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1163488256 bytes 2023-12-04T14:15:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1187004416 bytes 2023-12-04T14:15:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1197164544 bytes 2023-12-04T14:15:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1223992320 bytes 2023-12-04T14:15:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1282640896 bytes 2023-12-04T14:15:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1307131904 bytes 2023-12-04T14:15:01.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1384038400 bytes 2023-12-04T14:15:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1434086400 bytes 2023-12-04T14:15:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1438616576 bytes 2023-12-04T14:15:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1442566144 bytes 2023-12-04T14:15:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1512907776 bytes 2023-12-04T14:15:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1568659456 bytes 2023-12-04T14:15:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1626489856 bytes 2023-12-04T14:15:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1711954944 bytes 2023-12-04T14:15:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1712576512 bytes 2023-12-04T14:15:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1750117376 bytes 2023-12-04T14:15:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1765158912 bytes 2023-12-04T14:15:03.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1802928128 bytes 2023-12-04T14:15:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1840925696 bytes 2023-12-04T14:15:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 1851699200 bytes 2023-12-04T14:15:03.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 1894210560 bytes 2023-12-04T14:15:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:15:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:15:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:15:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:15:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-04T14:15:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:15:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:15:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:15:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:15:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:15:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:15:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:15:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:15:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:15:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:15:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:15:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:15:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:15:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:15:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:15:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:15:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:15:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:15:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+replaying 2023-12-04T14:15:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: replaying, {"bytes_per_second":53655552.0,"bytes_per_snapshot":536555520.0,"last_snapshot_bytes":536555520,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701699282,"remote_snapshot_timestamp":1701699282,"replay_state":"idle"} 2023-12-04T14:15:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: service: mirror.1 on smithi176 2023-12-04T14:15:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:14:59 2023-12-04T14:15:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:15:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:15:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+stopped 2023-12-04T14:15:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: local image is primary 2023-12-04T14:15:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:15:04 2023-12-04T14:15:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:15:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:15:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ description: .*' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:15:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":53655552.0,"bytes_per_snapshot":536555520.0,"last_snapshot_bytes":536555520,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701699282,"remote_snapshot_timestamp":1701699282,"replay_state":"idle"} 2023-12-04T14:15:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:15:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:15:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^up+' 2023-12-04T14:15:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:15:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:15:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.1 on smithi176 2023-12-04T14:15:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:15:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'last_update=2023-12-04 14:14:59' 2023-12-04T14:15:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-04 14:14:59' 2023-12-04T14:15:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:15:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'prev_last_update=2023-12-04 14:14:59' 2023-12-04T14:15:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-04T14:15:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror.mirror_status 2023-12-04T14:15:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:15:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror.mirror_status 2023-12-04T14:15:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-04T14:15:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:15:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local last_update state 2023-12-04T14:15:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-04T14:15:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ last_update='2023-12-04 14:14:59' 2023-12-04T14:15:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-04T14:15:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=up+replaying 2023-12-04T14:15:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:15:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep up+replaying 2023-12-04T14:15:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:15:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:15:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:15:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap9 2023-12-04T14:15:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap9 2023-12-04T14:15:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:15:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap9 2023-12-04T14:15:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:15:23.279 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8349196776827449) > weight (0.25) 2023-12-04T14:15:23.280 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4961055233976742) > weight (0.25) 2023-12-04T14:15:23.280 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-04T14:15:23.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-04T14:15:23.281 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-04T14:15:23.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi176.stderr:2023-12-04T14:15:23.280+0000 7fc779303640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 91125) UID: 1000 2023-12-04T14:15:27.282 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-04T14:15:27.282 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:15:27.282 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-04T14:15:27.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-04T14:15:27.283 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-04T14:15:27.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-04T14:15:27.286 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-12-04T14:15:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-04T14:15:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-04T14:15:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:15:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:15:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:15:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:15:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-04T14:15:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:15:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:15:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:15:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-04T14:15:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:15:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:15:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:15:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:15:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:15:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:15:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:15:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:15:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:15:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:15:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:15:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:15:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:15:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:15:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:15:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:15:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:15:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:15:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:15:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:15:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:15:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:15:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:15:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:15:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:15:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:15:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:15:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:15:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:15:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:15:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:15:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:15:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:15:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:15:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:15:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:15:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:15:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:15:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:15:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:15:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:15:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:15:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:15:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:15:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:15:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:15:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:15:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:15:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:15:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:15:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:15:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:15:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:15:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:15:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:15:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:15:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:15:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:15:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:15:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:15:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:15:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:15:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-test.status 2023-12-04T14:15:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-test.status 2023-12-04T14:15:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:15:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:15:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.status 2023-12-04T14:15:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.status 2023-12-04T14:15:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:15:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:15:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-04T14:15:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:15:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-04T14:15:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 37 2023-12-04T14:15:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:15:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:15:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:15:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-04T14:15:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:15:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:15:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=37 2023-12-04T14:15:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:15:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:15:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:15:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:15:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:15:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:15:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:15:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:15:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:15:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:15:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:15:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:15:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:15:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 37 = '' 2023-12-04T14:15:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:15:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:15:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:15:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:15:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:15:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:15:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:15:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:15:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:15:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:15:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:15:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:15:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 37 = '' 2023-12-04T14:15:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:15:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:15:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:15:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:15:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:15:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:15:36.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:15:36.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:15:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:15:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:15:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:15:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:15:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 37 = '' 2023-12-04T14:15:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:15:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:15:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:15:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:15:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:15:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:15:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:15:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=37 2023-12-04T14:15:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 37 = 37 2023-12-04T14:15:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:15:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-12-04T14:15:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap9 2023-12-04T14:15:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:15:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:15:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:15:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'test': 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 512 MiB in 128 objects 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 18 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10ad877be9cc 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10ad877be9cc 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:14:52 2023 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:15:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:15:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:15:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:15:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:15:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:15:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:15:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snap8-clean ']' 2023-12-04T14:15:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-04T14:15:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap8-clean 2023-12-04T14:15:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:15:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:15:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:15:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:15:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:15:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:15:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:15:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:15:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:15:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:15:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:15:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap9 2023-12-04T14:15:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap9 2023-12-04T14:15:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:15:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:15:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:15:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:15:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:15:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:15:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:15:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:15:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:15:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:15:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:15:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ clean_snap_name=snap9-clean 2023-12-04T14:15:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-12-04T14:15:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:15:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap9-clean 2023-12-04T14:15:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-12-04T14:15:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:15:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror test 2023-12-04T14:15:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:15:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:15:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=8 2023-12-04T14:15:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:15:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-04T14:15:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 586752 bytes 2023-12-04T14:15:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 101287936 bytes 2023-12-04T14:15:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 148039680 bytes 2023-12-04T14:15:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 194522112 bytes 2023-12-04T14:15:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 242511872 bytes 2023-12-04T14:15:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 270064640 bytes 2023-12-04T14:15:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 309558272 bytes 2023-12-04T14:15:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 368040960 bytes 2023-12-04T14:15:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 417580032 bytes 2023-12-04T14:15:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 467666944 bytes 2023-12-04T14:15:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 586881024 bytes 2023-12-04T14:15:44.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 593799168 bytes 2023-12-04T14:15:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 635987968 bytes 2023-12-04T14:15:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 668357632 bytes 2023-12-04T14:15:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 668965888 bytes 2023-12-04T14:15:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 690332672 bytes 2023-12-04T14:15:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 800736256 bytes 2023-12-04T14:15:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 833013760 bytes 2023-12-04T14:15:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 863367168 bytes 2023-12-04T14:15:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 899181568 bytes 2023-12-04T14:15:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 974112768 bytes 2023-12-04T14:15:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 993937408 bytes 2023-12-04T14:15:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1001213952 bytes 2023-12-04T14:15:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1049264128 bytes 2023-12-04T14:15:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1086718976 bytes 2023-12-04T14:15:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1149134848 bytes 2023-12-04T14:15:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1182292992 bytes 2023-12-04T14:15:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1225739264 bytes 2023-12-04T14:15:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1293932544 bytes 2023-12-04T14:15:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1306313728 bytes 2023-12-04T14:15:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1315448832 bytes 2023-12-04T14:15:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1363089408 bytes 2023-12-04T14:15:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1388597248 bytes 2023-12-04T14:15:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1398301696 bytes 2023-12-04T14:15:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1465761792 bytes 2023-12-04T14:15:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1553768448 bytes 2023-12-04T14:15:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1688016896 bytes 2023-12-04T14:15:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1690622976 bytes 2023-12-04T14:15:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1718967296 bytes 2023-12-04T14:15:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1765857280 bytes 2023-12-04T14:15:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1783907328 bytes 2023-12-04T14:15:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1817185280 bytes 2023-12-04T14:15:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1817825280 bytes 2023-12-04T14:15:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1837288448 bytes 2023-12-04T14:15:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1892430848 bytes 2023-12-04T14:15:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1956754432 bytes 2023-12-04T14:15:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1963359232 bytes 2023-12-04T14:15:50.500 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-04T14:15:50.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-04T14:15:50.500 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-04T14:15:50.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi176.stderr:2023-12-04T14:15:50.499+0000 7f237abbe640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 94983) UID: 1000 2023-12-04T14:15:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2027445248 bytes 2023-12-04T14:15:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:15:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:15:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:15:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:15:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-04T14:15:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:15:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:15:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:15:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:15:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:15:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-04T14:15:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:15:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local description_pattern= 2023-12-04T14:15:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local service_pattern= 2023-12-04T14:15:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-04T14:15:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-04T14:15:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:15:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:15:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:15:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-04T14:15:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:15:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:test: 2023-12-04T14:15:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: global_id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:15:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+replaying 2023-12-04T14:15:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536555520.0,"last_snapshot_bytes":536555520,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701699282,"remote_snapshot_timestamp":1701699282,"replay_state":"idle"} 2023-12-04T14:15:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: service: mirror.1 on smithi176 2023-12-04T14:15:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:15:29 2023-12-04T14:15:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: peer_sites: 2023-12-04T14:15:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: name: cluster2 2023-12-04T14:15:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: state: up+stopped 2023-12-04T14:15:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: description: local image is primary 2023-12-04T14:15:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: last_update: 2023-12-04 14:15:35 2023-12-04T14:15:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:15:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:15:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^ description: .*' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:15:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536555520.0,"last_snapshot_bytes":536555520,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1701699282,"remote_snapshot_timestamp":1701699282,"replay_state":"idle"} 2023-12-04T14:15:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:15:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:15:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep '^up+' 2023-12-04T14:15:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:15:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:15:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.1 on smithi176 2023-12-04T14:15:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.mirror_status 2023-12-04T14:15:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'last_update=2023-12-04 14:15:29' 2023-12-04T14:15:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-04 14:15:29' 2023-12-04T14:15:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:15:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state_pattern=up+replaying 2023-12-04T14:15:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local 'prev_last_update=2023-12-04 14:15:29' 2023-12-04T14:15:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-04T14:15:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror.mirror_status 2023-12-04T14:15:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:15:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror.mirror_status 2023-12-04T14:15:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-04T14:15:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:15:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local last_update state 2023-12-04T14:15:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-04T14:15:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ last_update='2023-12-04 14:15:29' 2023-12-04T14:15:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-04T14:15:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=up+replaying 2023-12-04T14:15:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo up+replaying 2023-12-04T14:15:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep up+replaying 2023-12-04T14:15:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:up+replaying 2023-12-04T14:15:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:15:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:15:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap10 2023-12-04T14:15:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap10 2023-12-04T14:15:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:15:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:15:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:15:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap10 2023-12-04T14:15:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:15:52.501 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-04T14:15:52.501 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:15:52.501 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-04T14:15:52.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-04T14:15:52.501 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-04T14:15:52.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-04T14:15:52.504 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-12-04T14:16:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-04T14:16:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-04T14:16:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:16:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:16:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:16:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-04T14:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-04T14:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:16:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:16:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:16:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:16:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:16:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:16:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:16:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:16:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:16:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:16:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:16:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:16:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:16:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:16:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:16:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:16:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:16:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:16:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:16:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:16:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:16:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:16:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:16:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:16:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:16:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:16:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:16:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:16:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:16:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:16:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:16:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:16:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:16:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:16:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:16:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:16:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:16:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:16:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:16:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:16:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:16:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:16:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-04T14:16:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:16:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:16:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:16:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:16:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-04T14:16:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:16:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:16:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-test.status 2023-12-04T14:16:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-test.status 2023-12-04T14:16:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:16:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:16:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.status 2023-12-04T14:16:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.status 2023-12-04T14:16:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:16:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-04T14:16:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:16:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-04T14:16:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 41 2023-12-04T14:16:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:16:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:16:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:16:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-04T14:16:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:16:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:16:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=41 2023-12-04T14:16:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:16:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:16:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:16:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:16:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:16:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:16:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:16:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:16:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:16:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:16:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:16:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 41 = '' 2023-12-04T14:16:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:16:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:16:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:16:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:16:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:16:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:16:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:16:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:16:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:16:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:16:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 41 = '' 2023-12-04T14:16:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:16:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:16:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:16:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:16:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:16:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:16:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:16:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:16:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:16:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:16:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 41 = '' 2023-12-04T14:16:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:16:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:16:25.749 INFO:tasks.ceph.cluster1.osd.1.smithi146.stderr:2023-12-04T14:16:25.747+0000 7f0c25848640 -1 reset not still connected to 0x55e9a1aa6750 2023-12-04T14:16:25.749 INFO:tasks.ceph.cluster1.osd.1.smithi146.stderr:2023-12-04T14:16:25.747+0000 7f0c25848640 -1 reset not still connected to 0x55e9a1aa6f70 2023-12-04T14:16:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:16:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:16:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:16:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=41 2023-12-04T14:16:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 41 = 41 2023-12-04T14:16:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-12-04T14:16:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap10 2023-12-04T14:16:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:16:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:16:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:16:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-12-04T14:16:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'test': 2023-12-04T14:16:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 512 MiB in 128 objects 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 20 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10ad877be9cc 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10ad877be9cc 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:14:52 2023 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:16:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:16:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snap9-clean ']' 2023-12-04T14:16:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-04T14:16:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap9-clean 2023-12-04T14:16:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap10 2023-12-04T14:16:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap10 2023-12-04T14:16:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ clean_snap_name=snap10-clean 2023-12-04T14:16:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-12-04T14:16:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:16:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap10-clean 2023-12-04T14:16:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-12-04T14:16:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-04T14:16:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-04T14:16:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:16:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:16:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:16:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:16:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-04T14:16:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:16:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:16:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:16:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-04T14:16:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:16:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:16:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:16:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:16:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:16:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:16:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:16:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:16:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:16:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:16:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:16:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-04T14:16:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:16:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:16:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:16:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:16:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:16:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:16:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:16:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:16:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:16:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:16:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:16:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:16:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:16:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:16:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:16:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:16:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:16:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:16:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:16:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:16:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-04T14:16:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-04T14:16:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:16:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:16:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:16:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-04T14:16:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:16:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:16:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:16:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:16:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:16:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:16:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:16:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:16:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:16:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:16:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:16:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:16:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:16:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:16:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:16:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:16:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:16:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:16:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:16:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:16:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-04T14:16:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:16:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:16:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:16:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:16:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:16:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:16:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:16:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-04T14:16:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:16:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:16:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:16:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:16:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-04T14:16:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:16:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:16:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-test.status 2023-12-04T14:16:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-test.status 2023-12-04T14:16:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:16:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:16:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-test.status 2023-12-04T14:16:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-test.status 2023-12-04T14:16:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:16:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-04T14:16:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:16:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-04T14:16:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 44 2023-12-04T14:16:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:16:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:16:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.status 2023-12-04T14:16:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-04T14:16:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:16:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:16:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=44 2023-12-04T14:16:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:16:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:16:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:16:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:16:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:16:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:16:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:16:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:16:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:16:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:16:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:16:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 44 = '' 2023-12-04T14:16:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:16:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:16:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.status 2023-12-04T14:16:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-04T14:16:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:16:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:16:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=44 2023-12-04T14:16:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 44 = 44 2023-12-04T14:16:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-12-04T14:16:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:16:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap10-clean 2023-12-04T14:16:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:16:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:16:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:16:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-12-04T14:16:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'test': 2023-12-04T14:16:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 512 MiB in 128 objects 2023-12-04T14:16:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:16:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 21 2023-12-04T14:16:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10ad877be9cc 2023-12-04T14:16:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10ad877be9cc 2023-12-04T14:16:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:16:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:16:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:16:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:16:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:16:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:16:28 2023 2023-12-04T14:16:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:08:06 2023 2023-12-04T14:16:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:16:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:16:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:16:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7263b59e-bedf-4a34-b292-66544734e417 2023-12-04T14:16:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:16:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 1 10 2023-12-04T14:16:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:16:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap1 2023-12-04T14:16:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap1 2023-12-04T14:16:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap1 2023-12-04T14:16:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:36.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:36.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap1-clean 2023-12-04T14:16:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-04T14:16:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap1-clean 2023-12-04T14:16:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:16:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap2 2023-12-04T14:16:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap2 2023-12-04T14:16:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap2 2023-12-04T14:16:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap2-clean 2023-12-04T14:16:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-04T14:16:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap2-clean 2023-12-04T14:16:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:16:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap3 2023-12-04T14:16:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap3 2023-12-04T14:16:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap3 2023-12-04T14:16:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap3-clean 2023-12-04T14:16:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-04T14:16:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap3-clean 2023-12-04T14:16:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:16:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap4 2023-12-04T14:16:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap4 2023-12-04T14:16:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap4 2023-12-04T14:16:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap4-clean 2023-12-04T14:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-04T14:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap4-clean 2023-12-04T14:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:16:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap5 2023-12-04T14:16:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap5 2023-12-04T14:16:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap5 2023-12-04T14:16:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap5-clean 2023-12-04T14:16:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-04T14:16:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap5-clean 2023-12-04T14:16:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:51.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:16:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap6 2023-12-04T14:16:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap6 2023-12-04T14:16:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap6 2023-12-04T14:16:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap6-clean 2023-12-04T14:16:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-04T14:16:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap6-clean 2023-12-04T14:16:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:16:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap7 2023-12-04T14:16:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap7 2023-12-04T14:16:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap7 2023-12-04T14:16:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:58.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:16:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap7-clean 2023-12-04T14:16:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-04T14:16:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:16:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:16:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap7-clean 2023-12-04T14:16:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:16:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:16:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:16:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:16:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap8 2023-12-04T14:17:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap8 2023-12-04T14:17:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap8 2023-12-04T14:17:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:17:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:17:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:17:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:17:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:17:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap8-clean 2023-12-04T14:17:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-04T14:17:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap8-clean 2023-12-04T14:17:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:17:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:17:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:17:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:17:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:17:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap9 2023-12-04T14:17:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap9 2023-12-04T14:17:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap9 2023-12-04T14:17:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:17:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:17:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:17:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:17:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:17:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap9-clean 2023-12-04T14:17:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-04T14:17:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap9-clean 2023-12-04T14:17:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:17:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:17:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:17:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:17:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:17:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap10 2023-12-04T14:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap10 2023-12-04T14:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap10 2023-12-04T14:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:17:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:17:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:08.287 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-04T14:17:08.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-04T14:17:08.288 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-04T14:17:08.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi176.stderr:2023-12-04T14:17:08.287+0000 7f93abe7f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 93907) UID: 1000 2023-12-04T14:17:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:17:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap10-clean 2023-12-04T14:17:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror test snap10-clean 2023-12-04T14:17:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap10-clean 2023-12-04T14:17:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:17:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:17:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export 2023-12-04T14:17:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:17:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-04T14:17:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-test.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-test.export 2023-12-04T14:17:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 1 10 2023-12-04T14:17:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap1 2023-12-04T14:17:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-12-04T14:17:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap1 2023-12-04T14:17:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-12-04T14:17:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap1-clean 2023-12-04T14:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-12-04T14:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap1-clean 2023-12-04T14:17:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-12-04T14:17:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap2 2023-12-04T14:17:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-12-04T14:17:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap2 2023-12-04T14:17:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-12-04T14:17:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap2-clean 2023-12-04T14:17:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-12-04T14:17:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap2-clean 2023-12-04T14:17:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-12-04T14:17:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap3 2023-12-04T14:17:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-12-04T14:17:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap3 2023-12-04T14:17:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-12-04T14:17:14.289 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-04T14:17:14.289 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:17:14.289 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-04T14:17:14.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-04T14:17:14.290 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-04T14:17:14.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-04T14:17:14.292 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2023-12-04T14:17:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap3-clean 2023-12-04T14:17:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-12-04T14:17:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap3-clean 2023-12-04T14:17:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-12-04T14:17:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap4 2023-12-04T14:17:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-12-04T14:17:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap4 2023-12-04T14:17:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-12-04T14:17:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap4-clean 2023-12-04T14:17:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-12-04T14:17:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap4-clean 2023-12-04T14:17:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-12-04T14:17:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap5 2023-12-04T14:17:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-12-04T14:17:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap5 2023-12-04T14:17:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-12-04T14:17:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap5-clean 2023-12-04T14:17:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-12-04T14:17:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap5-clean 2023-12-04T14:17:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-12-04T14:17:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap6 2023-12-04T14:17:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-12-04T14:17:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap6 2023-12-04T14:17:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-12-04T14:17:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap6-clean 2023-12-04T14:17:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-12-04T14:17:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap6-clean 2023-12-04T14:17:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-12-04T14:17:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap7 2023-12-04T14:17:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-12-04T14:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap7 2023-12-04T14:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-12-04T14:17:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap7-clean 2023-12-04T14:17:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-12-04T14:17:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap7-clean 2023-12-04T14:17:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-12-04T14:17:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap8 2023-12-04T14:17:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-12-04T14:17:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap8 2023-12-04T14:17:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-12-04T14:17:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap8-clean 2023-12-04T14:17:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-12-04T14:17:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap8-clean 2023-12-04T14:17:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-12-04T14:17:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap9 2023-12-04T14:17:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-12-04T14:17:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap9 2023-12-04T14:17:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-12-04T14:17:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap9-clean 2023-12-04T14:17:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-12-04T14:17:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap9-clean 2023-12-04T14:17:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-12-04T14:17:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 10` 2023-12-04T14:17:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap10 2023-12-04T14:17:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-12-04T14:17:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap10 2023-12-04T14:17:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-12-04T14:17:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap10-clean 2023-12-04T14:17:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-12-04T14:17:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap=snap10-clean 2023-12-04T14:17:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-12-04T14:17:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing snap: 100% complete...done. 2023-12-04T14:17:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_image_retry cluster2 mirror test 2023-12-04T14:17:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-04T14:17:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0 2023-12-04T14:17:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ remove_image cluster2 mirror test 2023-12-04T14:17:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-12-04T14:17:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-12-04T14:17:29.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr:2023-12-04T14:17:29.754+0000 7f08373af640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559a96500820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-04T14:17:29.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr:2023-12-04T14:17:29.754+0000 7f083e3bd640 -1 rbd::mirror::ImageReplayer: 0x559a9630c280 [3/7263b59e-bedf-4a34-b292-66544734e417] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-04T14:17:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-04T14:17:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-12-04T14:17:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:17:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=deleted 2023-12-04T14:17:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image_id= 2023-12-04T14:17:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:17:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n '' 2023-12-04T14:17:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ get_image_id cluster1 mirror test 2023-12-04T14:17:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:17:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local pool=mirror 2023-12-04T14:17:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local image=test 2023-12-04T14:17:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-04T14:17:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-04T14:17:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-04T14:17:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image_id= 2023-12-04T14:17:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-04T14:17:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.1 2023-12-04T14:17:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_present cluster1 mirror test deleted '' 2023-12-04T14:17:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:17:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=test 2023-12-04T14:17:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=deleted 2023-12-04T14:17:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image_id= 2023-12-04T14:17:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=deleted 2023-12-04T14:17:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_image_id 2023-12-04T14:17:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ get_image_id cluster1 mirror test 2023-12-04T14:17:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:17:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local pool=mirror 2023-12-04T14:17:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local image=test 2023-12-04T14:17:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-04T14:17:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-04T14:17:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-04T14:17:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_image_id= 2023-12-04T14:17:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n '' 2023-12-04T14:17:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test deleted = deleted 2023-12-04T14:17:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ testlog 'TEST: create many images' 2023-12-04T14:17:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee -a /tmp/tmp.rDwoQpiP9p/rbd-mirror.test.log 2023-12-04T14:17:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ date '+%F %T' 2023-12-04T14:17:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04 14:17:30 TEST: create many images 2023-12-04T14:17:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:2023-12-04 14:17:30 TEST: create many images 2023-12-04T14:17:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ snap_name=snap 2023-12-04T14:17:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 1 50 2023-12-04T14:17:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:17:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_1 2023-12-04T14:17:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-12-04T14:17:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:17:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:17:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:17:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_1 128M 2023-12-04T14:17:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:17:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:17:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:17:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:17:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_1 2023-12-04T14:17:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:17:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:17:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-12-04T14:17:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:17:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:17:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-04T14:17:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:17:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-04T14:17:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_1': 2023-12-04T14:17:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:17:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:17:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 125e16ce5757 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.125e16ce5757 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:17:31 2023 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:17:31 2023 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:17:31 2023 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: b928065e-ca9b-41a2-9e4f-db3d8351e631 2023-12-04T14:17:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:17:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:17:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-04T14:17:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:17:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:17:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=8 2023-12-04T14:17:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:17:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:17:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 19535872 bytes 2023-12-04T14:17:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 122318848 bytes 2023-12-04T14:17:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 162748416 bytes 2023-12-04T14:17:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 233518080 bytes 2023-12-04T14:17:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 263960576 bytes 2023-12-04T14:17:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 265680896 bytes 2023-12-04T14:17:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 316342272 bytes 2023-12-04T14:17:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 372514816 bytes 2023-12-04T14:17:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 409330688 bytes 2023-12-04T14:17:33.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 442946560 bytes 2023-12-04T14:17:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 487574528 bytes 2023-12-04T14:17:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 551186432 bytes 2023-12-04T14:17:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 570707968 bytes 2023-12-04T14:17:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 629749760 bytes 2023-12-04T14:17:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 681494528 bytes 2023-12-04T14:17:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 701693952 bytes 2023-12-04T14:17:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 738188288 bytes 2023-12-04T14:17:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 744903680 bytes 2023-12-04T14:17:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 841773056 bytes 2023-12-04T14:17:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 885879808 bytes 2023-12-04T14:17:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 929583104 bytes 2023-12-04T14:17:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 960101376 bytes 2023-12-04T14:17:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1023005696 bytes 2023-12-04T14:17:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1023609856 bytes 2023-12-04T14:17:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1069237248 bytes 2023-12-04T14:17:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1108235264 bytes 2023-12-04T14:17:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1141165056 bytes 2023-12-04T14:17:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1207100416 bytes 2023-12-04T14:17:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1264356352 bytes 2023-12-04T14:17:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1354854400 bytes 2023-12-04T14:17:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1426483200 bytes 2023-12-04T14:17:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1470510080 bytes 2023-12-04T14:17:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1478097920 bytes 2023-12-04T14:17:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1595659264 bytes 2023-12-04T14:17:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1635297280 bytes 2023-12-04T14:17:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1696020480 bytes 2023-12-04T14:17:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1767595008 bytes 2023-12-04T14:17:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1818569728 bytes 2023-12-04T14:17:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1880433664 bytes 2023-12-04T14:17:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1928886272 bytes 2023-12-04T14:17:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1950584832 bytes 2023-12-04T14:17:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1971316736 bytes 2023-12-04T14:17:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1981599744 bytes 2023-12-04T14:17:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 2043372544 bytes 2023-12-04T14:17:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 2097020928 bytes 2023-12-04T14:17:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:17:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:17:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:17:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:17:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_2 2023-12-04T14:17:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-12-04T14:17:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:17:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:17:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:17:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_2 128M 2023-12-04T14:17:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:17:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:17:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:17:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:17:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_2 2023-12-04T14:17:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:17:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:17:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-12-04T14:17:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:17:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:17:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-04T14:17:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:17:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-04T14:17:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_2': 2023-12-04T14:17:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:17:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:17:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:17:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 1266b67028ce 2023-12-04T14:17:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.1266b67028ce 2023-12-04T14:17:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:17:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:17:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:17:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:17:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:17:39 2023 2023-12-04T14:17:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:17:39 2023 2023-12-04T14:17:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:17:39 2023 2023-12-04T14:17:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:17:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:17:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 9fe30a7e-3975-4be8-9ddc-ffdbf8f787e1 2023-12-04T14:17:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:17:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:17:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-04T14:17:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:17:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:17:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=12 2023-12-04T14:17:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:17:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:17:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 58654720 bytes 2023-12-04T14:17:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 78076928 bytes 2023-12-04T14:17:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 102808576 bytes 2023-12-04T14:17:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 133456896 bytes 2023-12-04T14:17:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 182096896 bytes 2023-12-04T14:17:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 182717440 bytes 2023-12-04T14:17:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 233980928 bytes 2023-12-04T14:17:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 271980544 bytes 2023-12-04T14:17:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 297324544 bytes 2023-12-04T14:17:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 343284736 bytes 2023-12-04T14:17:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 373346304 bytes 2023-12-04T14:17:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 428831744 bytes 2023-12-04T14:17:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 526161920 bytes 2023-12-04T14:17:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 608054272 bytes 2023-12-04T14:17:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 691309568 bytes 2023-12-04T14:17:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 749717504 bytes 2023-12-04T14:17:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 753120256 bytes 2023-12-04T14:17:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 795215872 bytes 2023-12-04T14:17:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 820098048 bytes 2023-12-04T14:17:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 864401408 bytes 2023-12-04T14:17:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 894745600 bytes 2023-12-04T14:17:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 941334528 bytes 2023-12-04T14:17:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 953767936 bytes 2023-12-04T14:17:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1055066112 bytes 2023-12-04T14:17:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1068521472 bytes 2023-12-04T14:17:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1153489920 bytes 2023-12-04T14:17:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1169808384 bytes 2023-12-04T14:17:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1194356736 bytes 2023-12-04T14:17:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1264046080 bytes 2023-12-04T14:17:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1321883648 bytes 2023-12-04T14:17:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1362153472 bytes 2023-12-04T14:17:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1404758016 bytes 2023-12-04T14:17:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1473963008 bytes 2023-12-04T14:17:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1546456064 bytes 2023-12-04T14:17:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1572807680 bytes 2023-12-04T14:17:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1576187904 bytes 2023-12-04T14:17:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1630409728 bytes 2023-12-04T14:17:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1671173120 bytes 2023-12-04T14:17:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1696215040 bytes 2023-12-04T14:17:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1733731328 bytes 2023-12-04T14:17:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1772441600 bytes 2023-12-04T14:17:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1798586368 bytes 2023-12-04T14:17:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1880682496 bytes 2023-12-04T14:17:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1945912320 bytes 2023-12-04T14:17:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 2067783680 bytes 2023-12-04T14:17:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 2106613760 bytes 2023-12-04T14:17:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2137749504 bytes 2023-12-04T14:17:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2163600384 bytes 2023-12-04T14:17:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2182929408 bytes 2023-12-04T14:17:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2249619456 bytes 2023-12-04T14:17:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2329162752 bytes 2023-12-04T14:17:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2348491776 bytes 2023-12-04T14:17:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2387809280 bytes 2023-12-04T14:17:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2419915776 bytes 2023-12-04T14:17:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2445540352 bytes 2023-12-04T14:17:49.506 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.42491788171776057) > weight (0.25) 2023-12-04T14:17:49.506 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-04T14:17:49.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-04T14:17:49.506 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-04T14:17:49.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi176.stderr:2023-12-04T14:17:49.506+0000 7f5b7ba51640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 95941) UID: 1000 2023-12-04T14:17:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2529786880 bytes 2023-12-04T14:17:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2556719104 bytes 2023-12-04T14:17:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2571591680 bytes 2023-12-04T14:17:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2620377088 bytes 2023-12-04T14:17:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2690323456 bytes 2023-12-04T14:17:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2690957312 bytes 2023-12-04T14:17:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2740584448 bytes 2023-12-04T14:17:50.741 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2818414592 bytes 2023-12-04T14:17:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2852981760 bytes 2023-12-04T14:17:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2883164160 bytes 2023-12-04T14:17:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2927626240 bytes 2023-12-04T14:17:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2973416448 bytes 2023-12-04T14:17:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 3027363840 bytes 2023-12-04T14:17:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 3044276224 bytes 2023-12-04T14:17:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 3044907008 bytes 2023-12-04T14:17:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 3103634432 bytes 2023-12-04T14:17:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:17:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:17:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:17:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_3 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_3 128M 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:17:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:17:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:17:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:17:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:17:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_3 2023-12-04T14:17:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-12-04T14:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-04T14:17:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:17:52.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-04T14:17:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_3': 2023-12-04T14:17:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:17:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:17:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:17:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 126f96d668b8 2023-12-04T14:17:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.126f96d668b8 2023-12-04T14:17:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:17:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:17:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:17:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:17:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:17:52 2023 2023-12-04T14:17:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:17:52 2023 2023-12-04T14:17:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:17:52 2023 2023-12-04T14:17:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:17:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:17:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: d5429052-81ab-46d6-ab3d-16695051435d 2023-12-04T14:17:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:17:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:17:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-04T14:17:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:17:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:17:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:17:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:17:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=10 2023-12-04T14:17:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:17:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:17:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 13991936 bytes 2023-12-04T14:17:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 74926080 bytes 2023-12-04T14:17:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 94997504 bytes 2023-12-04T14:17:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 154447872 bytes 2023-12-04T14:17:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 176412672 bytes 2023-12-04T14:17:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 192962560 bytes 2023-12-04T14:17:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 274512896 bytes 2023-12-04T14:17:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 328879104 bytes 2023-12-04T14:17:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 381386752 bytes 2023-12-04T14:17:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 389659648 bytes 2023-12-04T14:17:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 475046912 bytes 2023-12-04T14:17:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 500005888 bytes 2023-12-04T14:17:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 529243136 bytes 2023-12-04T14:17:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 571702272 bytes 2023-12-04T14:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 591004672 bytes 2023-12-04T14:17:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 676167680 bytes 2023-12-04T14:17:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 725028864 bytes 2023-12-04T14:17:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 729957376 bytes 2023-12-04T14:17:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 812703744 bytes 2023-12-04T14:17:56.507 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-04T14:17:56.507 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:17:56.508 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-04T14:17:56.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-04T14:17:56.508 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-04T14:17:56.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-04T14:17:56.511 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2023-12-04T14:17:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 854180864 bytes 2023-12-04T14:17:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 884296704 bytes 2023-12-04T14:17:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 942276608 bytes 2023-12-04T14:17:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 980718592 bytes 2023-12-04T14:17:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1047087104 bytes 2023-12-04T14:17:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1088901120 bytes 2023-12-04T14:17:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1122294784 bytes 2023-12-04T14:17:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1156744192 bytes 2023-12-04T14:17:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1187829760 bytes 2023-12-04T14:17:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1204120576 bytes 2023-12-04T14:17:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1287165952 bytes 2023-12-04T14:17:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1351400448 bytes 2023-12-04T14:17:58.667 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1377333248 bytes 2023-12-04T14:17:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1409010688 bytes 2023-12-04T14:17:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1456047104 bytes 2023-12-04T14:17:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1507226624 bytes 2023-12-04T14:17:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1525019648 bytes 2023-12-04T14:17:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1585543168 bytes 2023-12-04T14:17:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1640287232 bytes 2023-12-04T14:17:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1675223040 bytes 2023-12-04T14:18:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1726964736 bytes 2023-12-04T14:18:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1833339904 bytes 2023-12-04T14:18:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1871888384 bytes 2023-12-04T14:18:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1900180480 bytes 2023-12-04T14:18:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1908079616 bytes 2023-12-04T14:18:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1935473664 bytes 2023-12-04T14:18:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1986487296 bytes 2023-12-04T14:18:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2025277440 bytes 2023-12-04T14:18:01.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2100786176 bytes 2023-12-04T14:18:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2125752320 bytes 2023-12-04T14:18:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2166347776 bytes 2023-12-04T14:18:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2194437120 bytes 2023-12-04T14:18:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2232426496 bytes 2023-12-04T14:18:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2286433280 bytes 2023-12-04T14:18:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2346858496 bytes 2023-12-04T14:18:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2441748480 bytes 2023-12-04T14:18:02.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2487154688 bytes 2023-12-04T14:18:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2517046272 bytes 2023-12-04T14:18:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:18:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:18:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:18:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:18:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:18:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_4 2023-12-04T14:18:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-12-04T14:18:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:18:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:18:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:18:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_4 128M 2023-12-04T14:18:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:18:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:18:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:18:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:18:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_4 2023-12-04T14:18:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:18:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:18:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:18:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-12-04T14:18:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:18:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:18:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-04T14:18:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:18:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-04T14:18:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_4': 2023-12-04T14:18:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:18:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:18:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:18:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 12804eef7eb2 2023-12-04T14:18:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.12804eef7eb2 2023-12-04T14:18:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:18:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:18:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:18:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:18:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:03 2023 2023-12-04T14:18:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:18:03 2023 2023-12-04T14:18:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:03 2023 2023-12-04T14:18:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:18:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:18:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 60498ab7-7310-4365-aa39-30c7019acf9f 2023-12-04T14:18:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:18:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:18:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-04T14:18:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:18:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:18:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=11 2023-12-04T14:18:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:18:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:18:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 32097280 bytes 2023-12-04T14:18:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 76713984 bytes 2023-12-04T14:18:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 137456640 bytes 2023-12-04T14:18:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 157821952 bytes 2023-12-04T14:18:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 190850048 bytes 2023-12-04T14:18:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 230135808 bytes 2023-12-04T14:18:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 319264768 bytes 2023-12-04T14:18:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 359860224 bytes 2023-12-04T14:18:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 436582400 bytes 2023-12-04T14:18:05.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 461864960 bytes 2023-12-04T14:18:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 489447424 bytes 2023-12-04T14:18:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 522844160 bytes 2023-12-04T14:18:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 573739008 bytes 2023-12-04T14:18:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 664092672 bytes 2023-12-04T14:18:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 691080192 bytes 2023-12-04T14:18:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 716789760 bytes 2023-12-04T14:18:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 720125952 bytes 2023-12-04T14:18:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 748932096 bytes 2023-12-04T14:18:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 757608448 bytes 2023-12-04T14:18:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 810624000 bytes 2023-12-04T14:18:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 880189440 bytes 2023-12-04T14:18:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 911753216 bytes 2023-12-04T14:18:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 960323584 bytes 2023-12-04T14:18:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1002339328 bytes 2023-12-04T14:18:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1061023744 bytes 2023-12-04T14:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1095813120 bytes 2023-12-04T14:18:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1138913280 bytes 2023-12-04T14:18:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1179020288 bytes 2023-12-04T14:18:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1208040448 bytes 2023-12-04T14:18:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1238441984 bytes 2023-12-04T14:18:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1278326784 bytes 2023-12-04T14:18:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1303973888 bytes 2023-12-04T14:18:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1325835264 bytes 2023-12-04T14:18:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1366181888 bytes 2023-12-04T14:18:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1414079488 bytes 2023-12-04T14:18:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1551255552 bytes 2023-12-04T14:18:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1578717184 bytes 2023-12-04T14:18:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1622813696 bytes 2023-12-04T14:18:10.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1706618880 bytes 2023-12-04T14:18:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1765262336 bytes 2023-12-04T14:18:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1791187968 bytes 2023-12-04T14:18:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1837398016 bytes 2023-12-04T14:18:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1892980736 bytes 2023-12-04T14:18:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1893607424 bytes 2023-12-04T14:18:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1919425536 bytes 2023-12-04T14:18:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1967968256 bytes 2023-12-04T14:18:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1972219904 bytes 2023-12-04T14:18:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1996016640 bytes 2023-12-04T14:18:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2061780992 bytes 2023-12-04T14:18:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2100386816 bytes 2023-12-04T14:18:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2129189888 bytes 2023-12-04T14:18:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2144571392 bytes 2023-12-04T14:18:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2178259968 bytes 2023-12-04T14:18:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2286863360 bytes 2023-12-04T14:18:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2349905920 bytes 2023-12-04T14:18:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2373891072 bytes 2023-12-04T14:18:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2437784576 bytes 2023-12-04T14:18:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2444746752 bytes 2023-12-04T14:18:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2549989376 bytes 2023-12-04T14:18:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2577005568 bytes 2023-12-04T14:18:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2613511168 bytes 2023-12-04T14:18:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2635096064 bytes 2023-12-04T14:18:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2718742528 bytes 2023-12-04T14:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_5 2023-12-04T14:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-12-04T14:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:18:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:18:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_5 128M 2023-12-04T14:18:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:18:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:18:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:18:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:18:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_5 2023-12-04T14:18:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:18:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:18:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:18:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-12-04T14:18:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:18:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:18:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-04T14:18:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:18:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-04T14:18:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_5': 2023-12-04T14:18:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:18:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:18:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:18:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 128946eb3ae3 2023-12-04T14:18:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.128946eb3ae3 2023-12-04T14:18:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:18:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:18:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:18:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:18:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:15 2023 2023-12-04T14:18:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:18:15 2023 2023-12-04T14:18:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:15 2023 2023-12-04T14:18:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:18:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:18:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 90bc0c25-b4ac-4b4a-8f12-bbe6c286b2cd 2023-12-04T14:18:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:18:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:18:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-04T14:18:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:18:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:18:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=8 2023-12-04T14:18:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:18:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:18:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 46010368 bytes 2023-12-04T14:18:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 98550784 bytes 2023-12-04T14:18:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 117788672 bytes 2023-12-04T14:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 156597248 bytes 2023-12-04T14:18:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 213742592 bytes 2023-12-04T14:18:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 275563520 bytes 2023-12-04T14:18:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 286282752 bytes 2023-12-04T14:18:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 328144896 bytes 2023-12-04T14:18:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 340819968 bytes 2023-12-04T14:18:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 379550720 bytes 2023-12-04T14:18:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 410474496 bytes 2023-12-04T14:18:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 463633408 bytes 2023-12-04T14:18:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 488474624 bytes 2023-12-04T14:18:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 613472256 bytes 2023-12-04T14:18:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 683236352 bytes 2023-12-04T14:18:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 716872704 bytes 2023-12-04T14:18:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 725112832 bytes 2023-12-04T14:18:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 762479616 bytes 2023-12-04T14:18:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 774756352 bytes 2023-12-04T14:18:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 798825472 bytes 2023-12-04T14:18:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 816069632 bytes 2023-12-04T14:18:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 833673216 bytes 2023-12-04T14:18:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 887003136 bytes 2023-12-04T14:18:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 929444864 bytes 2023-12-04T14:18:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 953614336 bytes 2023-12-04T14:18:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 985809920 bytes 2023-12-04T14:18:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 986427392 bytes 2023-12-04T14:18:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 993555456 bytes 2023-12-04T14:18:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1029957632 bytes 2023-12-04T14:18:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1073613824 bytes 2023-12-04T14:18:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1110306816 bytes 2023-12-04T14:18:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1203941376 bytes 2023-12-04T14:18:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1219246080 bytes 2023-12-04T14:18:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1240026112 bytes 2023-12-04T14:18:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1277641728 bytes 2023-12-04T14:18:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1345732608 bytes 2023-12-04T14:18:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1409869824 bytes 2023-12-04T14:18:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1443552256 bytes 2023-12-04T14:18:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1473125376 bytes 2023-12-04T14:18:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1530738688 bytes 2023-12-04T14:18:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1533631488 bytes 2023-12-04T14:18:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1569660928 bytes 2023-12-04T14:18:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1637150720 bytes 2023-12-04T14:18:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1685746688 bytes 2023-12-04T14:18:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1736249344 bytes 2023-12-04T14:18:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1789057024 bytes 2023-12-04T14:18:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1812653056 bytes 2023-12-04T14:18:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1846943744 bytes 2023-12-04T14:18:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 1908743168 bytes 2023-12-04T14:18:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:18:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:18:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:18:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:18:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:18:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_6 2023-12-04T14:18:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-12-04T14:18:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:18:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:18:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:18:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_6 128M 2023-12-04T14:18:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:18:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:18:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:18:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:18:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_6 2023-12-04T14:18:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:18:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:18:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:18:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-12-04T14:18:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:18:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:18:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-04T14:18:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:18:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-04T14:18:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_6': 2023-12-04T14:18:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:18:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:18:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:18:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 129324ecd84f 2023-12-04T14:18:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.129324ecd84f 2023-12-04T14:18:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:18:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:18:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:18:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:18:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:23 2023 2023-12-04T14:18:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:18:23 2023 2023-12-04T14:18:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:23 2023 2023-12-04T14:18:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:18:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:18:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 481d8ed8-0c1d-4f88-8ce7-bbcfd2363766 2023-12-04T14:18:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:18:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:18:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-04T14:18:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:18:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:18:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=9 2023-12-04T14:18:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:18:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:18:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 25569280 bytes 2023-12-04T14:18:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 78952448 bytes 2023-12-04T14:18:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 124343296 bytes 2023-12-04T14:18:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 213603328 bytes 2023-12-04T14:18:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 264798208 bytes 2023-12-04T14:18:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 325332992 bytes 2023-12-04T14:18:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 346432512 bytes 2023-12-04T14:18:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 417886208 bytes 2023-12-04T14:18:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 443932672 bytes 2023-12-04T14:18:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 599269376 bytes 2023-12-04T14:18:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 623191040 bytes 2023-12-04T14:18:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 647037952 bytes 2023-12-04T14:18:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 684991488 bytes 2023-12-04T14:18:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 710691840 bytes 2023-12-04T14:18:27.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 755999744 bytes 2023-12-04T14:18:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 762070016 bytes 2023-12-04T14:18:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 803720192 bytes 2023-12-04T14:18:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 830810112 bytes 2023-12-04T14:18:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 892895232 bytes 2023-12-04T14:18:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 949749760 bytes 2023-12-04T14:18:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 990501888 bytes 2023-12-04T14:18:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 999750656 bytes 2023-12-04T14:18:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1027394560 bytes 2023-12-04T14:18:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1107644416 bytes 2023-12-04T14:18:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1148947456 bytes 2023-12-04T14:18:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1201640448 bytes 2023-12-04T14:18:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1243559936 bytes 2023-12-04T14:18:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1283638272 bytes 2023-12-04T14:18:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1426079744 bytes 2023-12-04T14:18:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1471553536 bytes 2023-12-04T14:18:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1503733760 bytes 2023-12-04T14:18:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1528898560 bytes 2023-12-04T14:18:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1568258048 bytes 2023-12-04T14:18:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1591895040 bytes 2023-12-04T14:18:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1658301440 bytes 2023-12-04T14:18:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1718665216 bytes 2023-12-04T14:18:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1747335168 bytes 2023-12-04T14:18:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1774862336 bytes 2023-12-04T14:18:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1797699584 bytes 2023-12-04T14:18:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1843847168 bytes 2023-12-04T14:18:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1878052864 bytes 2023-12-04T14:18:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1878667264 bytes 2023-12-04T14:18:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1890453504 bytes 2023-12-04T14:18:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1938903040 bytes 2023-12-04T14:18:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1976304640 bytes 2023-12-04T14:18:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 2007786496 bytes 2023-12-04T14:18:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2083357696 bytes 2023-12-04T14:18:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2103396352 bytes 2023-12-04T14:18:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2129673216 bytes 2023-12-04T14:18:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2148998144 bytes 2023-12-04T14:18:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2231686144 bytes 2023-12-04T14:18:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2240257024 bytes 2023-12-04T14:18:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:18:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:18:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:18:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:18:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:18:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_7 2023-12-04T14:18:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-12-04T14:18:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:18:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:18:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:18:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_7 128M 2023-12-04T14:18:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:18:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:18:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:18:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:18:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_7 2023-12-04T14:18:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:18:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:18:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:18:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-12-04T14:18:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:18:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:18:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-04T14:18:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:18:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-04T14:18:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_7': 2023-12-04T14:18:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:18:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 129c8ad85a1a 2023-12-04T14:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.129c8ad85a1a 2023-12-04T14:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:33 2023 2023-12-04T14:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:18:33 2023 2023-12-04T14:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:33 2023 2023-12-04T14:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:18:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:18:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 431573ac-8014-4867-b7be-738ee54b8f15 2023-12-04T14:18:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:18:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:18:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-04T14:18:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:18:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:18:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=5 2023-12-04T14:18:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:18:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:18:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 80113664 bytes 2023-12-04T14:18:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 106795008 bytes 2023-12-04T14:18:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 141009920 bytes 2023-12-04T14:18:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 179389440 bytes 2023-12-04T14:18:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 226770944 bytes 2023-12-04T14:18:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 259024896 bytes 2023-12-04T14:18:35.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 314254336 bytes 2023-12-04T14:18:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 324546560 bytes 2023-12-04T14:18:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 348615680 bytes 2023-12-04T14:18:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 429043712 bytes 2023-12-04T14:18:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 506842112 bytes 2023-12-04T14:18:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 539741184 bytes 2023-12-04T14:18:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 583143424 bytes 2023-12-04T14:18:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 593145856 bytes 2023-12-04T14:18:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 648743936 bytes 2023-12-04T14:18:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 692844544 bytes 2023-12-04T14:18:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 711397376 bytes 2023-12-04T14:18:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 737373184 bytes 2023-12-04T14:18:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 762641408 bytes 2023-12-04T14:18:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 913138688 bytes 2023-12-04T14:18:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 948091904 bytes 2023-12-04T14:18:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1008840704 bytes 2023-12-04T14:18:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1044159488 bytes 2023-12-04T14:18:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1107153920 bytes 2023-12-04T14:18:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1137798144 bytes 2023-12-04T14:18:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1145175040 bytes 2023-12-04T14:18:39.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1204763648 bytes 2023-12-04T14:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1243506688 bytes 2023-12-04T14:18:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1263724544 bytes 2023-12-04T14:18:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:18:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:18:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:18:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:18:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:18:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_8 2023-12-04T14:18:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-12-04T14:18:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:18:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:18:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:18:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_8 128M 2023-12-04T14:18:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:18:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:18:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:18:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:18:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_8 2023-12-04T14:18:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:18:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:18:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:18:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-12-04T14:18:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:18:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:18:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-04T14:18:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:18:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-04T14:18:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_8': 2023-12-04T14:18:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:18:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:18:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:18:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 12a491577b29 2023-12-04T14:18:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.12a491577b29 2023-12-04T14:18:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:18:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:18:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:18:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:18:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:39 2023 2023-12-04T14:18:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:18:39 2023 2023-12-04T14:18:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:39 2023 2023-12-04T14:18:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:18:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:18:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 14168f82-d9d2-41fc-99ed-f1205d8ef156 2023-12-04T14:18:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:18:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:18:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-04T14:18:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:18:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:18:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=9 2023-12-04T14:18:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:18:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:18:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 87808000 bytes 2023-12-04T14:18:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 95788032 bytes 2023-12-04T14:18:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 151984128 bytes 2023-12-04T14:18:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 152597504 bytes 2023-12-04T14:18:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 199735296 bytes 2023-12-04T14:18:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 287748096 bytes 2023-12-04T14:18:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 347130880 bytes 2023-12-04T14:18:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 372673536 bytes 2023-12-04T14:18:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 429982720 bytes 2023-12-04T14:18:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 436852736 bytes 2023-12-04T14:18:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 483211264 bytes 2023-12-04T14:18:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 505165824 bytes 2023-12-04T14:18:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 535753728 bytes 2023-12-04T14:18:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 561835008 bytes 2023-12-04T14:18:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 566141952 bytes 2023-12-04T14:18:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 590827520 bytes 2023-12-04T14:18:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 638529536 bytes 2023-12-04T14:18:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 708745216 bytes 2023-12-04T14:18:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 742910976 bytes 2023-12-04T14:18:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 817645568 bytes 2023-12-04T14:18:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 866826240 bytes 2023-12-04T14:18:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 927821824 bytes 2023-12-04T14:18:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 947708928 bytes 2023-12-04T14:18:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 998196224 bytes 2023-12-04T14:18:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1035867136 bytes 2023-12-04T14:18:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1074444288 bytes 2023-12-04T14:18:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1090899968 bytes 2023-12-04T14:18:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1174660096 bytes 2023-12-04T14:18:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1241949184 bytes 2023-12-04T14:18:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1293782016 bytes 2023-12-04T14:18:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1370549248 bytes 2023-12-04T14:18:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1415345152 bytes 2023-12-04T14:18:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1468331008 bytes 2023-12-04T14:18:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1486716928 bytes 2023-12-04T14:18:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1521422336 bytes 2023-12-04T14:18:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1566833664 bytes 2023-12-04T14:18:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1592665088 bytes 2023-12-04T14:18:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1673904128 bytes 2023-12-04T14:18:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1743578112 bytes 2023-12-04T14:18:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1804660736 bytes 2023-12-04T14:18:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1835355136 bytes 2023-12-04T14:18:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1835976704 bytes 2023-12-04T14:18:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1921737728 bytes 2023-12-04T14:18:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1952845824 bytes 2023-12-04T14:18:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1980270592 bytes 2023-12-04T14:18:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 2012252160 bytes 2023-12-04T14:18:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2073655296 bytes 2023-12-04T14:18:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2105092096 bytes 2023-12-04T14:18:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2150422528 bytes 2023-12-04T14:18:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2235992064 bytes 2023-12-04T14:18:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2320168960 bytes 2023-12-04T14:18:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:18:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:18:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:18:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:18:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:18:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_9 2023-12-04T14:18:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_9 128M 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:18:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:18:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:18:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:18:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:18:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_9 2023-12-04T14:18:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:18:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:18:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:18:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-12-04T14:18:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:18:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:18:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-04T14:18:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:18:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-04T14:18:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_9': 2023-12-04T14:18:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:18:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:18:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:18:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 12acea28894f 2023-12-04T14:18:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.12acea28894f 2023-12-04T14:18:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:18:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:18:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:18:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:18:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:49 2023 2023-12-04T14:18:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:18:49 2023 2023-12-04T14:18:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:49 2023 2023-12-04T14:18:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:18:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:18:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: fc8c6534-5c92-48ae-b43e-099c17bc6417 2023-12-04T14:18:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:18:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:18:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-04T14:18:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:18:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:18:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:18:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:18:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=13 2023-12-04T14:18:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:18:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:18:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 46124032 bytes 2023-12-04T14:18:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 82697216 bytes 2023-12-04T14:18:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 117698560 bytes 2023-12-04T14:18:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 131664896 bytes 2023-12-04T14:18:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 146667520 bytes 2023-12-04T14:18:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 183729152 bytes 2023-12-04T14:18:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 229516288 bytes 2023-12-04T14:18:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 243996672 bytes 2023-12-04T14:18:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 264881152 bytes 2023-12-04T14:18:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 281699328 bytes 2023-12-04T14:18:51.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 302164992 bytes 2023-12-04T14:18:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 314470400 bytes 2023-12-04T14:18:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 347423744 bytes 2023-12-04T14:18:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 392330240 bytes 2023-12-04T14:18:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 392932352 bytes 2023-12-04T14:18:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 469208064 bytes 2023-12-04T14:18:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 469804032 bytes 2023-12-04T14:18:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 505306112 bytes 2023-12-04T14:18:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 540220416 bytes 2023-12-04T14:18:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 565466112 bytes 2023-12-04T14:18:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 568400896 bytes 2023-12-04T14:18:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 580400128 bytes 2023-12-04T14:18:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 643857408 bytes 2023-12-04T14:18:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 644484096 bytes 2023-12-04T14:18:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 648901632 bytes 2023-12-04T14:18:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 682503168 bytes 2023-12-04T14:18:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 749519872 bytes 2023-12-04T14:18:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 800586752 bytes 2023-12-04T14:18:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 849818624 bytes 2023-12-04T14:18:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 861272064 bytes 2023-12-04T14:18:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 918158336 bytes 2023-12-04T14:18:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 975482880 bytes 2023-12-04T14:18:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1045264384 bytes 2023-12-04T14:18:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1084950528 bytes 2023-12-04T14:18:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1098180608 bytes 2023-12-04T14:18:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1103312896 bytes 2023-12-04T14:18:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1120922624 bytes 2023-12-04T14:18:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1146094592 bytes 2023-12-04T14:18:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1157068800 bytes 2023-12-04T14:18:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1177309184 bytes 2023-12-04T14:18:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1286952960 bytes 2023-12-04T14:18:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1338877952 bytes 2023-12-04T14:18:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1344019456 bytes 2023-12-04T14:18:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1414158336 bytes 2023-12-04T14:18:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1438360576 bytes 2023-12-04T14:18:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1492422656 bytes 2023-12-04T14:18:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1520259072 bytes 2023-12-04T14:18:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1551559680 bytes 2023-12-04T14:18:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 1636583424 bytes 2023-12-04T14:18:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 1637201920 bytes 2023-12-04T14:18:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 1654809600 bytes 2023-12-04T14:18:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 1700978688 bytes 2023-12-04T14:18:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 1723564032 bytes 2023-12-04T14:18:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 1737437184 bytes 2023-12-04T14:18:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 1782228992 bytes 2023-12-04T14:18:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 1831143424 bytes 2023-12-04T14:18:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 1898143744 bytes 2023-12-04T14:18:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 1942907904 bytes 2023-12-04T14:18:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2020264960 bytes 2023-12-04T14:18:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2056446976 bytes 2023-12-04T14:18:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2085254144 bytes 2023-12-04T14:18:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2139058176 bytes 2023-12-04T14:18:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2219262976 bytes 2023-12-04T14:19:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2254343168 bytes 2023-12-04T14:19:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2263193600 bytes 2023-12-04T14:19:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2284655616 bytes 2023-12-04T14:19:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2340042752 bytes 2023-12-04T14:19:01.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2397396992 bytes 2023-12-04T14:19:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2406072320 bytes 2023-12-04T14:19:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 2413783040 bytes 2023-12-04T14:19:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 2504937472 bytes 2023-12-04T14:19:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 2527549440 bytes 2023-12-04T14:19:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7300 IOs, 2538370048 bytes 2023-12-04T14:19:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7400 IOs, 2620334080 bytes 2023-12-04T14:19:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7500 IOs, 2653510656 bytes 2023-12-04T14:19:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7600 IOs, 2697238528 bytes 2023-12-04T14:19:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7700 IOs, 2752049152 bytes 2023-12-04T14:19:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7800 IOs, 2823115776 bytes 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_10 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_10 128M 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:19:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:19:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_10 2023-12-04T14:19:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:19:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:19:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:19:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-12-04T14:19:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:19:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:19:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-04T14:19:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:19:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-04T14:19:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_10': 2023-12-04T14:19:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:19:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:19:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:19:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 12b55cca9063 2023-12-04T14:19:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.12b55cca9063 2023-12-04T14:19:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:19:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:19:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:19:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:19:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:03 2023 2023-12-04T14:19:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:19:03 2023 2023-12-04T14:19:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:03 2023 2023-12-04T14:19:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:19:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:19:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 0d9ffbfe-5f0b-4261-82aa-d3c625dda130 2023-12-04T14:19:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-04T14:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:19:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=8 2023-12-04T14:19:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:19:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:19:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 44661760 bytes 2023-12-04T14:19:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 97261568 bytes 2023-12-04T14:19:04.512 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-04T14:19:04.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-04T14:19:04.513 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-04T14:19:04.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi176.stderr:2023-12-04T14:19:04.512+0000 7f2cf4e20640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 97600) UID: 1000 2023-12-04T14:19:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 113254400 bytes 2023-12-04T14:19:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 178616320 bytes 2023-12-04T14:19:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 203144192 bytes 2023-12-04T14:19:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 229493760 bytes 2023-12-04T14:19:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 243792896 bytes 2023-12-04T14:19:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 255045632 bytes 2023-12-04T14:19:05.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 275007488 bytes 2023-12-04T14:19:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 354463744 bytes 2023-12-04T14:19:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 398883840 bytes 2023-12-04T14:19:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 404257792 bytes 2023-12-04T14:19:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 427157504 bytes 2023-12-04T14:19:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 476838912 bytes 2023-12-04T14:19:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 518852608 bytes 2023-12-04T14:19:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 578743296 bytes 2023-12-04T14:19:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 646337536 bytes 2023-12-04T14:19:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 668416000 bytes 2023-12-04T14:19:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 714406912 bytes 2023-12-04T14:19:07.293 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5922542234755711) > weight (0.25) 2023-12-04T14:19:07.293 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8016327594617911) > weight (0.25) 2023-12-04T14:19:07.293 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-04T14:19:07.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-04T14:19:07.294 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-04T14:19:07.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi176.stderr:2023-12-04T14:19:07.294+0000 7ff00404a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 97104) UID: 1000 2023-12-04T14:19:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 753537024 bytes 2023-12-04T14:19:07.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 853863424 bytes 2023-12-04T14:19:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 854491136 bytes 2023-12-04T14:19:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 910949376 bytes 2023-12-04T14:19:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 966074368 bytes 2023-12-04T14:19:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 995922944 bytes 2023-12-04T14:19:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1041656832 bytes 2023-12-04T14:19:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1051510784 bytes 2023-12-04T14:19:08.514 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-04T14:19:08.514 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:19:08.514 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-04T14:19:08.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-04T14:19:08.515 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-04T14:19:08.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-04T14:19:08.517 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2023-12-04T14:19:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1080324096 bytes 2023-12-04T14:19:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1099839488 bytes 2023-12-04T14:19:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1162487808 bytes 2023-12-04T14:19:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1181957120 bytes 2023-12-04T14:19:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1248786432 bytes 2023-12-04T14:19:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1281187840 bytes 2023-12-04T14:19:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1386712064 bytes 2023-12-04T14:19:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1437872128 bytes 2023-12-04T14:19:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1474035712 bytes 2023-12-04T14:19:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1503457280 bytes 2023-12-04T14:19:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1539575808 bytes 2023-12-04T14:19:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1592922112 bytes 2023-12-04T14:19:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1630213120 bytes 2023-12-04T14:19:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1647529984 bytes 2023-12-04T14:19:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1684001792 bytes 2023-12-04T14:19:11.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1707702272 bytes 2023-12-04T14:19:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1744090112 bytes 2023-12-04T14:19:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1765392384 bytes 2023-12-04T14:19:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1784489984 bytes 2023-12-04T14:19:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1834142720 bytes 2023-12-04T14:19:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1899199488 bytes 2023-12-04T14:19:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:19:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:19:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:19:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:19:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:19:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_11 2023-12-04T14:19:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-12-04T14:19:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_11 128M 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:19:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:19:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:19:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_11 2023-12-04T14:19:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:19:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:19:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:19:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-12-04T14:19:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:19:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:19:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-12-04T14:19:12.294 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-04T14:19:12.294 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:19:12.295 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-04T14:19:12.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-04T14:19:12.295 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-04T14:19:12.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-04T14:19:12.298 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2023-12-04T14:19:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:19:13.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_11': 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 12c4189e1f39 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.12c4189e1f39 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:12 2023 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:19:12 2023 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:12 2023 2023-12-04T14:19:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:19:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:19:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 99047729-2de2-4c91-90dc-1e2bfe4f2b5f 2023-12-04T14:19:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:19:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:19:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_11 2023-12-04T14:19:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:19:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:19:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=9 2023-12-04T14:19:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:19:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:19:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 23890944 bytes 2023-12-04T14:19:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 56730624 bytes 2023-12-04T14:19:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 127533056 bytes 2023-12-04T14:19:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 130806784 bytes 2023-12-04T14:19:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 225295360 bytes 2023-12-04T14:19:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 285192192 bytes 2023-12-04T14:19:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 323411968 bytes 2023-12-04T14:19:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 361399296 bytes 2023-12-04T14:19:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 385125376 bytes 2023-12-04T14:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 436833280 bytes 2023-12-04T14:19:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 496297984 bytes 2023-12-04T14:19:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 538334208 bytes 2023-12-04T14:19:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 626339840 bytes 2023-12-04T14:19:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 651065344 bytes 2023-12-04T14:19:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 723803136 bytes 2023-12-04T14:19:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 739576832 bytes 2023-12-04T14:19:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 792072192 bytes 2023-12-04T14:19:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 839800832 bytes 2023-12-04T14:19:16.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 896618496 bytes 2023-12-04T14:19:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 898571264 bytes 2023-12-04T14:19:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 968057856 bytes 2023-12-04T14:19:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 993422336 bytes 2023-12-04T14:19:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1042953216 bytes 2023-12-04T14:19:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1071324160 bytes 2023-12-04T14:19:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1103230976 bytes 2023-12-04T14:19:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1144283136 bytes 2023-12-04T14:19:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1191414784 bytes 2023-12-04T14:19:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1220173824 bytes 2023-12-04T14:19:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1285091328 bytes 2023-12-04T14:19:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1363501056 bytes 2023-12-04T14:19:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1394984960 bytes 2023-12-04T14:19:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1417692160 bytes 2023-12-04T14:19:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1497915392 bytes 2023-12-04T14:19:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1503460352 bytes 2023-12-04T14:19:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1504479232 bytes 2023-12-04T14:19:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1505341440 bytes 2023-12-04T14:19:19.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1534861312 bytes 2023-12-04T14:19:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1552288768 bytes 2023-12-04T14:19:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1579143168 bytes 2023-12-04T14:19:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1625752576 bytes 2023-12-04T14:19:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1669082112 bytes 2023-12-04T14:19:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1716977664 bytes 2023-12-04T14:19:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1738007552 bytes 2023-12-04T14:19:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1795914752 bytes 2023-12-04T14:19:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1816497152 bytes 2023-12-04T14:19:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1839095808 bytes 2023-12-04T14:19:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1858822144 bytes 2023-12-04T14:19:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1859432448 bytes 2023-12-04T14:19:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 1885232128 bytes 2023-12-04T14:19:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 1995909120 bytes 2023-12-04T14:19:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2054334464 bytes 2023-12-04T14:19:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2058529792 bytes 2023-12-04T14:19:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2098324480 bytes 2023-12-04T14:19:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2136641536 bytes 2023-12-04T14:19:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:19:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:19:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:19:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:19:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:19:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_12 2023-12-04T14:19:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-12-04T14:19:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:19:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:19:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_12 128M 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_12 2023-12-04T14:19:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:19:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:19:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:19:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-12-04T14:19:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:19:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:19:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-12-04T14:19:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:19:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-12-04T14:19:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_12': 2023-12-04T14:19:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:19:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:19:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 12dabbd3031f 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.12dabbd3031f 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:22 2023 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:19:22 2023 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:22 2023 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 5f322e9d-b400-4d44-81e0-ffdd216398b2 2023-12-04T14:19:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:19:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:19:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_12 2023-12-04T14:19:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:19:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:19:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=12 2023-12-04T14:19:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:19:23.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:19:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 8815616 bytes 2023-12-04T14:19:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 45689856 bytes 2023-12-04T14:19:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 46296064 bytes 2023-12-04T14:19:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 96109568 bytes 2023-12-04T14:19:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 185532416 bytes 2023-12-04T14:19:24.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 237087744 bytes 2023-12-04T14:19:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 320916480 bytes 2023-12-04T14:19:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 355103744 bytes 2023-12-04T14:19:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 389695488 bytes 2023-12-04T14:19:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 421472256 bytes 2023-12-04T14:19:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 473940992 bytes 2023-12-04T14:19:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 567606272 bytes 2023-12-04T14:19:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 607622144 bytes 2023-12-04T14:19:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 658814976 bytes 2023-12-04T14:19:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 729723904 bytes 2023-12-04T14:19:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 739081216 bytes 2023-12-04T14:19:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 793735168 bytes 2023-12-04T14:19:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 812263424 bytes 2023-12-04T14:19:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 871142400 bytes 2023-12-04T14:19:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 900769792 bytes 2023-12-04T14:19:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 949941248 bytes 2023-12-04T14:19:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 963796992 bytes 2023-12-04T14:19:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1006703616 bytes 2023-12-04T14:19:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1059629056 bytes 2023-12-04T14:19:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1144663040 bytes 2023-12-04T14:19:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1177360384 bytes 2023-12-04T14:19:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1193152512 bytes 2023-12-04T14:19:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1273837568 bytes 2023-12-04T14:19:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1324443648 bytes 2023-12-04T14:19:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1376168960 bytes 2023-12-04T14:19:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1437301760 bytes 2023-12-04T14:19:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1501785088 bytes 2023-12-04T14:19:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1536119808 bytes 2023-12-04T14:19:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1572174848 bytes 2023-12-04T14:19:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1692631040 bytes 2023-12-04T14:19:30.033 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:19:30.031+0000 7f8e4ae66640 -1 reset not still connected to 0x55b1a168f930 2023-12-04T14:19:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1767958528 bytes 2023-12-04T14:19:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1781459968 bytes 2023-12-04T14:19:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1820254208 bytes 2023-12-04T14:19:30.542 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:19:30.540+0000 7f8e4ae66640 -1 reset not still connected to 0x55b1a168f930 2023-12-04T14:19:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1895205888 bytes 2023-12-04T14:19:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1917129728 bytes 2023-12-04T14:19:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1938814976 bytes 2023-12-04T14:19:31.091 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:19:31.090+0000 7f8e4ae66640 -1 reset not still connected to 0x55b1a168f930 2023-12-04T14:19:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 2000697344 bytes 2023-12-04T14:19:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 2048984064 bytes 2023-12-04T14:19:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 2049603584 bytes 2023-12-04T14:19:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 2068464640 bytes 2023-12-04T14:19:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 2115388416 bytes 2023-12-04T14:19:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2198009856 bytes 2023-12-04T14:19:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2219193344 bytes 2023-12-04T14:19:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2296258560 bytes 2023-12-04T14:19:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2375019520 bytes 2023-12-04T14:19:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2381509632 bytes 2023-12-04T14:19:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2425742336 bytes 2023-12-04T14:19:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2499991552 bytes 2023-12-04T14:19:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2501713920 bytes 2023-12-04T14:19:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2504591360 bytes 2023-12-04T14:19:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2506612736 bytes 2023-12-04T14:19:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2552240128 bytes 2023-12-04T14:19:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2578208768 bytes 2023-12-04T14:19:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2590312448 bytes 2023-12-04T14:19:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2651134976 bytes 2023-12-04T14:19:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2676389888 bytes 2023-12-04T14:19:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2731509760 bytes 2023-12-04T14:19:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2793017344 bytes 2023-12-04T14:19:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2795720704 bytes 2023-12-04T14:19:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2809577472 bytes 2023-12-04T14:19:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2864784384 bytes 2023-12-04T14:19:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2924495872 bytes 2023-12-04T14:19:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:19:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:19:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:19:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_13 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_13 128M 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:19:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:19:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:19:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:19:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_13 2023-12-04T14:19:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:19:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:19:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:19:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-12-04T14:19:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:19:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:19:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-12-04T14:19:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:19:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_13': 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 12e640f5e13 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.12e640f5e13 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:35 2023 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:19:35 2023 2023-12-04T14:19:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:35 2023 2023-12-04T14:19:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:19:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:19:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: ac478d96-184c-4eaa-8c62-f6f0e8288dff 2023-12-04T14:19:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:19:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:19:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_13 2023-12-04T14:19:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:19:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:19:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=7 2023-12-04T14:19:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:19:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:19:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 29264896 bytes 2023-12-04T14:19:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 71969792 bytes 2023-12-04T14:19:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 99184640 bytes 2023-12-04T14:19:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 133059584 bytes 2023-12-04T14:19:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 168415232 bytes 2023-12-04T14:19:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 214929408 bytes 2023-12-04T14:19:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 235619328 bytes 2023-12-04T14:19:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 236266496 bytes 2023-12-04T14:19:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 286961664 bytes 2023-12-04T14:19:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 340721664 bytes 2023-12-04T14:19:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 378563584 bytes 2023-12-04T14:19:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 449274880 bytes 2023-12-04T14:19:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 523719680 bytes 2023-12-04T14:19:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 612317184 bytes 2023-12-04T14:19:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 621664256 bytes 2023-12-04T14:19:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 626792448 bytes 2023-12-04T14:19:38.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 633300992 bytes 2023-12-04T14:19:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 683901952 bytes 2023-12-04T14:19:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 743895040 bytes 2023-12-04T14:19:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 794811392 bytes 2023-12-04T14:19:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 838994944 bytes 2023-12-04T14:19:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 871910400 bytes 2023-12-04T14:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 992983040 bytes 2023-12-04T14:19:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1055906816 bytes 2023-12-04T14:19:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1077741568 bytes 2023-12-04T14:19:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1097707520 bytes 2023-12-04T14:19:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1118157824 bytes 2023-12-04T14:19:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1157524480 bytes 2023-12-04T14:19:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1185820672 bytes 2023-12-04T14:19:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1251587072 bytes 2023-12-04T14:19:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1341363200 bytes 2023-12-04T14:19:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1368513536 bytes 2023-12-04T14:19:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1445664768 bytes 2023-12-04T14:19:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1481924608 bytes 2023-12-04T14:19:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1492001792 bytes 2023-12-04T14:19:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1585693696 bytes 2023-12-04T14:19:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1605046272 bytes 2023-12-04T14:19:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1638447104 bytes 2023-12-04T14:19:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:19:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:19:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:19:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:19:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:19:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_14 2023-12-04T14:19:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-12-04T14:19:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:19:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:19:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:19:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_14 128M 2023-12-04T14:19:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:19:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:19:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:19:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:19:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_14 2023-12-04T14:19:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:19:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:19:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:19:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-12-04T14:19:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:19:42.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:19:42.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-12-04T14:19:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:19:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-12-04T14:19:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_14': 2023-12-04T14:19:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:19:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:19:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:19:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 12f2b8567914 2023-12-04T14:19:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.12f2b8567914 2023-12-04T14:19:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:19:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:19:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:19:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:19:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:42 2023 2023-12-04T14:19:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:19:42 2023 2023-12-04T14:19:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:42 2023 2023-12-04T14:19:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:19:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:19:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 8794a2f7-791e-4d63-afdb-a1f297b316fa 2023-12-04T14:19:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:19:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:19:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_14 2023-12-04T14:19:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:19:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:19:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=14 2023-12-04T14:19:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:19:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:19:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 6135808 bytes 2023-12-04T14:19:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 54780928 bytes 2023-12-04T14:19:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 102977536 bytes 2023-12-04T14:19:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 135427072 bytes 2023-12-04T14:19:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 157195264 bytes 2023-12-04T14:19:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 204729344 bytes 2023-12-04T14:19:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 244494336 bytes 2023-12-04T14:19:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 266713088 bytes 2023-12-04T14:19:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 329265152 bytes 2023-12-04T14:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 386165760 bytes 2023-12-04T14:19:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 393835520 bytes 2023-12-04T14:19:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 425456640 bytes 2023-12-04T14:19:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 437304320 bytes 2023-12-04T14:19:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 466859008 bytes 2023-12-04T14:19:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 501884928 bytes 2023-12-04T14:19:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 528470016 bytes 2023-12-04T14:19:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 557087744 bytes 2023-12-04T14:19:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 585229312 bytes 2023-12-04T14:19:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 620858368 bytes 2023-12-04T14:19:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 676594688 bytes 2023-12-04T14:19:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 688217088 bytes 2023-12-04T14:19:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 738333696 bytes 2023-12-04T14:19:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 744911872 bytes 2023-12-04T14:19:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 771952640 bytes 2023-12-04T14:19:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 788253696 bytes 2023-12-04T14:19:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 818347008 bytes 2023-12-04T14:19:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 912444416 bytes 2023-12-04T14:19:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 954353664 bytes 2023-12-04T14:19:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 995713024 bytes 2023-12-04T14:19:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1092380672 bytes 2023-12-04T14:19:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1119540224 bytes 2023-12-04T14:19:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1146819584 bytes 2023-12-04T14:19:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1163129856 bytes 2023-12-04T14:19:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1188206592 bytes 2023-12-04T14:19:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1213701120 bytes 2023-12-04T14:19:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1221830656 bytes 2023-12-04T14:19:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1222438912 bytes 2023-12-04T14:19:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1223056384 bytes 2023-12-04T14:19:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1263947776 bytes 2023-12-04T14:19:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1313692672 bytes 2023-12-04T14:19:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1352617984 bytes 2023-12-04T14:19:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1437888512 bytes 2023-12-04T14:19:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1510921216 bytes 2023-12-04T14:19:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1547600896 bytes 2023-12-04T14:19:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1567170560 bytes 2023-12-04T14:19:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1622584320 bytes 2023-12-04T14:19:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1682458624 bytes 2023-12-04T14:19:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1709097984 bytes 2023-12-04T14:19:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 1758409728 bytes 2023-12-04T14:19:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 1813913600 bytes 2023-12-04T14:19:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 1915977728 bytes 2023-12-04T14:19:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 1972924416 bytes 2023-12-04T14:19:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 1980954624 bytes 2023-12-04T14:19:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2012670976 bytes 2023-12-04T14:19:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2037432320 bytes 2023-12-04T14:19:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2105351168 bytes 2023-12-04T14:19:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2169592832 bytes 2023-12-04T14:19:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2188493824 bytes 2023-12-04T14:19:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2233439232 bytes 2023-12-04T14:19:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2287567872 bytes 2023-12-04T14:19:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2288175104 bytes 2023-12-04T14:19:53.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2367774720 bytes 2023-12-04T14:19:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2403735552 bytes 2023-12-04T14:19:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2417697792 bytes 2023-12-04T14:19:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2430259200 bytes 2023-12-04T14:19:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2476292096 bytes 2023-12-04T14:19:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2501771264 bytes 2023-12-04T14:19:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2534716416 bytes 2023-12-04T14:19:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2541698048 bytes 2023-12-04T14:19:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 2597286912 bytes 2023-12-04T14:19:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 2633407488 bytes 2023-12-04T14:19:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 2662519808 bytes 2023-12-04T14:19:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7300 IOs, 2733868032 bytes 2023-12-04T14:19:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7400 IOs, 2797937664 bytes 2023-12-04T14:19:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7500 IOs, 2874449920 bytes 2023-12-04T14:19:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7600 IOs, 2911486976 bytes 2023-12-04T14:19:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7700 IOs, 2936501248 bytes 2023-12-04T14:19:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7800 IOs, 3011263488 bytes 2023-12-04T14:19:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7900 IOs, 3071554560 bytes 2023-12-04T14:19:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8000 IOs, 3101588480 bytes 2023-12-04T14:19:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8100 IOs, 3136464896 bytes 2023-12-04T14:19:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8200 IOs, 3184952320 bytes 2023-12-04T14:19:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:19:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:19:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:19:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:19:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:19:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_15 2023-12-04T14:19:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-12-04T14:19:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_15 128M 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:19:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:19:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:19:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:19:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_15 2023-12-04T14:19:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:19:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:19:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:19:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-12-04T14:19:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:19:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:19:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-12-04T14:19:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:19:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-12-04T14:19:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_15': 2023-12-04T14:19:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:19:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:19:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:19:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 12fe4c90759d 2023-12-04T14:19:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.12fe4c90759d 2023-12-04T14:19:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:19:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:19:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:19:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:19:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:57 2023 2023-12-04T14:19:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:19:57 2023 2023-12-04T14:19:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:57 2023 2023-12-04T14:19:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:19:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:19:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 0e39536e-cf2c-4b79-ae6d-a5d1ede7920c 2023-12-04T14:19:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:19:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:19:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_15 2023-12-04T14:19:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:19:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:19:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:19:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:19:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=8 2023-12-04T14:19:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:19:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:19:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 123039744 bytes 2023-12-04T14:19:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 167670784 bytes 2023-12-04T14:19:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 216481792 bytes 2023-12-04T14:19:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 271020032 bytes 2023-12-04T14:19:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 308118528 bytes 2023-12-04T14:19:59.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 328625152 bytes 2023-12-04T14:19:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 351147008 bytes 2023-12-04T14:19:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 395479040 bytes 2023-12-04T14:19:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 434313216 bytes 2023-12-04T14:19:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 479288320 bytes 2023-12-04T14:20:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 496381952 bytes 2023-12-04T14:20:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 532799488 bytes 2023-12-04T14:20:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 586729472 bytes 2023-12-04T14:20:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 683875328 bytes 2023-12-04T14:20:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 772425728 bytes 2023-12-04T14:20:01.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 809151488 bytes 2023-12-04T14:20:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 892424192 bytes 2023-12-04T14:20:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 919819264 bytes 2023-12-04T14:20:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 935181312 bytes 2023-12-04T14:20:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 962004992 bytes 2023-12-04T14:20:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 1016953856 bytes 2023-12-04T14:20:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1057723392 bytes 2023-12-04T14:20:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1094800384 bytes 2023-12-04T14:20:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1202965504 bytes 2023-12-04T14:20:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1272797184 bytes 2023-12-04T14:20:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1289423872 bytes 2023-12-04T14:20:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1312156672 bytes 2023-12-04T14:20:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1342972928 bytes 2023-12-04T14:20:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1359664128 bytes 2023-12-04T14:20:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1362204672 bytes 2023-12-04T14:20:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1413905408 bytes 2023-12-04T14:20:03.837 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:20:03.834+0000 7f4bfa770640 -1 reset not still connected to 0x56230f5b8f70 2023-12-04T14:20:03.837 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:20:03.834+0000 7f4bfa770640 -1 reset not still connected to 0x562310005380 2023-12-04T14:20:03.837 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:20:03.834+0000 7f4bfa770640 -1 reset not still connected to 0x56231a33f040 2023-12-04T14:20:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1491796992 bytes 2023-12-04T14:20:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1533776896 bytes 2023-12-04T14:20:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1572163584 bytes 2023-12-04T14:20:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1572777984 bytes 2023-12-04T14:20:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1639544832 bytes 2023-12-04T14:20:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1684623360 bytes 2023-12-04T14:20:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1706432512 bytes 2023-12-04T14:20:05.081 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1720568832 bytes 2023-12-04T14:20:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1785355264 bytes 2023-12-04T14:20:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1814035456 bytes 2023-12-04T14:20:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1871970304 bytes 2023-12-04T14:20:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1952892928 bytes 2023-12-04T14:20:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1978005504 bytes 2023-12-04T14:20:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:20:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:20:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:20:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:20:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:20:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_16 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_16 128M 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_16 2023-12-04T14:20:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:20:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:20:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:20:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-12-04T14:20:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:20:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:20:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-12-04T14:20:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:20:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_16': 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 130aa766c627 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.130aa766c627 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:06 2023 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:20:06 2023 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:06 2023 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:20:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:20:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 5854cc13-6872-45e6-8c82-645c028f7199 2023-12-04T14:20:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:20:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:20:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_16 2023-12-04T14:20:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:20:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:20:06.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=12 2023-12-04T14:20:06.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:20:06.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:20:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 53627904 bytes 2023-12-04T14:20:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 67502080 bytes 2023-12-04T14:20:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 93747200 bytes 2023-12-04T14:20:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 127233024 bytes 2023-12-04T14:20:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 184611840 bytes 2023-12-04T14:20:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 199290880 bytes 2023-12-04T14:20:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 302827520 bytes 2023-12-04T14:20:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 340085760 bytes 2023-12-04T14:20:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 351782912 bytes 2023-12-04T14:20:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 356230144 bytes 2023-12-04T14:20:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 409432064 bytes 2023-12-04T14:20:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 420602880 bytes 2023-12-04T14:20:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 471422976 bytes 2023-12-04T14:20:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 505993216 bytes 2023-12-04T14:20:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 522934272 bytes 2023-12-04T14:20:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 565292032 bytes 2023-12-04T14:20:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 577594368 bytes 2023-12-04T14:20:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 647814144 bytes 2023-12-04T14:20:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 682071040 bytes 2023-12-04T14:20:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 721453056 bytes 2023-12-04T14:20:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 748286976 bytes 2023-12-04T14:20:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 833498112 bytes 2023-12-04T14:20:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 858161152 bytes 2023-12-04T14:20:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 919876608 bytes 2023-12-04T14:20:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 971977728 bytes 2023-12-04T14:20:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1015340032 bytes 2023-12-04T14:20:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1016035328 bytes 2023-12-04T14:20:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1059460096 bytes 2023-12-04T14:20:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1139545088 bytes 2023-12-04T14:20:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1223234560 bytes 2023-12-04T14:20:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1270282240 bytes 2023-12-04T14:20:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1343873024 bytes 2023-12-04T14:20:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1400069120 bytes 2023-12-04T14:20:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1443407872 bytes 2023-12-04T14:20:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1481616384 bytes 2023-12-04T14:20:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1582763008 bytes 2023-12-04T14:20:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1633012736 bytes 2023-12-04T14:20:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1712200704 bytes 2023-12-04T14:20:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1738292224 bytes 2023-12-04T14:20:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1766505472 bytes 2023-12-04T14:20:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1768517632 bytes 2023-12-04T14:20:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1808312320 bytes 2023-12-04T14:20:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1828560896 bytes 2023-12-04T14:20:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1881564160 bytes 2023-12-04T14:20:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1900718080 bytes 2023-12-04T14:20:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1914628096 bytes 2023-12-04T14:20:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2020444160 bytes 2023-12-04T14:20:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2065626112 bytes 2023-12-04T14:20:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2090422272 bytes 2023-12-04T14:20:15.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2120421376 bytes 2023-12-04T14:20:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2136652800 bytes 2023-12-04T14:20:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2220103680 bytes 2023-12-04T14:20:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2245577728 bytes 2023-12-04T14:20:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2287971328 bytes 2023-12-04T14:20:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2330902528 bytes 2023-12-04T14:20:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2368908288 bytes 2023-12-04T14:20:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2382164992 bytes 2023-12-04T14:20:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2383448064 bytes 2023-12-04T14:20:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2425718784 bytes 2023-12-04T14:20:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2476986368 bytes 2023-12-04T14:20:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2535325696 bytes 2023-12-04T14:20:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2586039296 bytes 2023-12-04T14:20:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2627779584 bytes 2023-12-04T14:20:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2649356288 bytes 2023-12-04T14:20:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2697989120 bytes 2023-12-04T14:20:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2771540992 bytes 2023-12-04T14:20:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:20:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:20:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:20:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:20:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:20:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_17 2023-12-04T14:20:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-12-04T14:20:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:20:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:20:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:20:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_17 128M 2023-12-04T14:20:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:20:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:20:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:20:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:20:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_17 2023-12-04T14:20:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:20:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:20:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:20:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-12-04T14:20:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:20:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:20:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-12-04T14:20:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:20:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-12-04T14:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_17': 2023-12-04T14:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 1316d0632b20 2023-12-04T14:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.1316d0632b20 2023-12-04T14:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:18 2023 2023-12-04T14:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:20:18 2023 2023-12-04T14:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:18 2023 2023-12-04T14:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7bfe12f9-a769-4f59-bca6-8c32a93f3809 2023-12-04T14:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:20:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:20:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_17 2023-12-04T14:20:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:20:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:20:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=13 2023-12-04T14:20:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:20:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:20:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 92347392 bytes 2023-12-04T14:20:20.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 145459200 bytes 2023-12-04T14:20:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 159038464 bytes 2023-12-04T14:20:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 194529280 bytes 2023-12-04T14:20:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 257024000 bytes 2023-12-04T14:20:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 263505920 bytes 2023-12-04T14:20:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 265178112 bytes 2023-12-04T14:20:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 311150592 bytes 2023-12-04T14:20:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 353947648 bytes 2023-12-04T14:20:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 418062336 bytes 2023-12-04T14:20:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 487516160 bytes 2023-12-04T14:20:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 494894080 bytes 2023-12-04T14:20:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 495508480 bytes 2023-12-04T14:20:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 554300416 bytes 2023-12-04T14:20:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 583403520 bytes 2023-12-04T14:20:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 595173376 bytes 2023-12-04T14:20:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 632740864 bytes 2023-12-04T14:20:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 658547712 bytes 2023-12-04T14:20:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 677284864 bytes 2023-12-04T14:20:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 734026752 bytes 2023-12-04T14:20:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 781906944 bytes 2023-12-04T14:20:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 830262272 bytes 2023-12-04T14:20:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 871609344 bytes 2023-12-04T14:20:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 875820032 bytes 2023-12-04T14:20:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 911020032 bytes 2023-12-04T14:20:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 954914816 bytes 2023-12-04T14:20:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1001274368 bytes 2023-12-04T14:20:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1047455744 bytes 2023-12-04T14:20:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1086976000 bytes 2023-12-04T14:20:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1111433216 bytes 2023-12-04T14:20:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1194036224 bytes 2023-12-04T14:20:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1239271424 bytes 2023-12-04T14:20:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1262950400 bytes 2023-12-04T14:20:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1334130688 bytes 2023-12-04T14:20:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1379728384 bytes 2023-12-04T14:20:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1421051904 bytes 2023-12-04T14:20:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1476264960 bytes 2023-12-04T14:20:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1491379200 bytes 2023-12-04T14:20:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1534644224 bytes 2023-12-04T14:20:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1612891136 bytes 2023-12-04T14:20:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1647399936 bytes 2023-12-04T14:20:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1702780928 bytes 2023-12-04T14:20:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1750207488 bytes 2023-12-04T14:20:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1824617472 bytes 2023-12-04T14:20:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1854420992 bytes 2023-12-04T14:20:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1895770112 bytes 2023-12-04T14:20:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1981009920 bytes 2023-12-04T14:20:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2008515584 bytes 2023-12-04T14:20:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2026423296 bytes 2023-12-04T14:20:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2074684416 bytes 2023-12-04T14:20:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2104977408 bytes 2023-12-04T14:20:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2153270272 bytes 2023-12-04T14:20:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2218275840 bytes 2023-12-04T14:20:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2248525824 bytes 2023-12-04T14:20:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2282355712 bytes 2023-12-04T14:20:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2336572416 bytes 2023-12-04T14:20:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2426800128 bytes 2023-12-04T14:20:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2503215104 bytes 2023-12-04T14:20:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2521738240 bytes 2023-12-04T14:20:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2558429184 bytes 2023-12-04T14:20:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2622484480 bytes 2023-12-04T14:20:30.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2664345600 bytes 2023-12-04T14:20:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2722305024 bytes 2023-12-04T14:20:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2756790272 bytes 2023-12-04T14:20:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2787574784 bytes 2023-12-04T14:20:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2827585536 bytes 2023-12-04T14:20:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2877363200 bytes 2023-12-04T14:20:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2947449856 bytes 2023-12-04T14:20:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2960680960 bytes 2023-12-04T14:20:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 3031308288 bytes 2023-12-04T14:20:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 3074691072 bytes 2023-12-04T14:20:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 3075277824 bytes 2023-12-04T14:20:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:20:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:20:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:20:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:20:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_18 2023-12-04T14:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-12-04T14:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:20:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_18 128M 2023-12-04T14:20:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:20:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:20:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:20:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:20:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_18 2023-12-04T14:20:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:20:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:20:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:20:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-12-04T14:20:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:20:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:20:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-12-04T14:20:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:20:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_18': 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 1322687b04f3 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.1322687b04f3 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:32 2023 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:20:32 2023 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:32 2023 2023-12-04T14:20:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:20:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:20:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 9d431cf5-1ffc-4498-b3e0-5bba35366d30 2023-12-04T14:20:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:20:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:20:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_18 2023-12-04T14:20:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:20:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:20:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=13 2023-12-04T14:20:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:20:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:20:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 23321600 bytes 2023-12-04T14:20:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 29883392 bytes 2023-12-04T14:20:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 56616960 bytes 2023-12-04T14:20:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 77192192 bytes 2023-12-04T14:20:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 93701120 bytes 2023-12-04T14:20:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 120841216 bytes 2023-12-04T14:20:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 172729344 bytes 2023-12-04T14:20:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 212571136 bytes 2023-12-04T14:20:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 277670912 bytes 2023-12-04T14:20:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 304961536 bytes 2023-12-04T14:20:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 348022784 bytes 2023-12-04T14:20:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 373273600 bytes 2023-12-04T14:20:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 389433344 bytes 2023-12-04T14:20:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 426330112 bytes 2023-12-04T14:20:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 438311936 bytes 2023-12-04T14:20:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 489515008 bytes 2023-12-04T14:20:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 511975424 bytes 2023-12-04T14:20:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 546301952 bytes 2023-12-04T14:20:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 584314880 bytes 2023-12-04T14:20:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 597274624 bytes 2023-12-04T14:20:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 622803968 bytes 2023-12-04T14:20:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 680268800 bytes 2023-12-04T14:20:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 737935360 bytes 2023-12-04T14:20:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 791656448 bytes 2023-12-04T14:20:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 817664000 bytes 2023-12-04T14:20:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 859616256 bytes 2023-12-04T14:20:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 927822848 bytes 2023-12-04T14:20:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 973251584 bytes 2023-12-04T14:20:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 973876224 bytes 2023-12-04T14:20:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1055600640 bytes 2023-12-04T14:20:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1089110016 bytes 2023-12-04T14:20:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1112818688 bytes 2023-12-04T14:20:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1144438784 bytes 2023-12-04T14:20:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1210996736 bytes 2023-12-04T14:20:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1233758208 bytes 2023-12-04T14:20:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1272808448 bytes 2023-12-04T14:20:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1273435136 bytes 2023-12-04T14:20:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1296000000 bytes 2023-12-04T14:20:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1341759488 bytes 2023-12-04T14:20:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1403108352 bytes 2023-12-04T14:20:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1429507072 bytes 2023-12-04T14:20:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1500903424 bytes 2023-12-04T14:20:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1527911424 bytes 2023-12-04T14:20:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1588838400 bytes 2023-12-04T14:20:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1666167808 bytes 2023-12-04T14:20:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1706807296 bytes 2023-12-04T14:20:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1740871680 bytes 2023-12-04T14:20:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1783994368 bytes 2023-12-04T14:20:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 1829223424 bytes 2023-12-04T14:20:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 1829820416 bytes 2023-12-04T14:20:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 1892415488 bytes 2023-12-04T14:20:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 1921933312 bytes 2023-12-04T14:20:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 1984659456 bytes 2023-12-04T14:20:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2027322368 bytes 2023-12-04T14:20:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2065132544 bytes 2023-12-04T14:20:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2100102144 bytes 2023-12-04T14:20:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2115474432 bytes 2023-12-04T14:20:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2148134912 bytes 2023-12-04T14:20:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2196499456 bytes 2023-12-04T14:20:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2242102272 bytes 2023-12-04T14:20:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2266575872 bytes 2023-12-04T14:20:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2301080576 bytes 2023-12-04T14:20:43.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2388566016 bytes 2023-12-04T14:20:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2450295808 bytes 2023-12-04T14:20:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2471759872 bytes 2023-12-04T14:20:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2482012160 bytes 2023-12-04T14:20:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2516469760 bytes 2023-12-04T14:20:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2557339648 bytes 2023-12-04T14:20:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2621227008 bytes 2023-12-04T14:20:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 2638642176 bytes 2023-12-04T14:20:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 2704693248 bytes 2023-12-04T14:20:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 2787861504 bytes 2023-12-04T14:20:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7300 IOs, 2841034752 bytes 2023-12-04T14:20:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7400 IOs, 2891430912 bytes 2023-12-04T14:20:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7500 IOs, 2932627456 bytes 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_19 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:20:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_19 128M 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:20:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_19 2023-12-04T14:20:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:20:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:20:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:20:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-12-04T14:20:46.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:46.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:46.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:20:46.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:20:46.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-12-04T14:20:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:20:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-12-04T14:20:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_19': 2023-12-04T14:20:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:20:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:20:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:20:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 132e12b6dd36 2023-12-04T14:20:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.132e12b6dd36 2023-12-04T14:20:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:20:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:20:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:20:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:20:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:46 2023 2023-12-04T14:20:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:20:46 2023 2023-12-04T14:20:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:46 2023 2023-12-04T14:20:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:20:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:20:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 44e764dc-83c2-44b2-bca6-672e14a04ca5 2023-12-04T14:20:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:20:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:20:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_19 2023-12-04T14:20:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:20:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:20:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=7 2023-12-04T14:20:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:20:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:20:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 24737792 bytes 2023-12-04T14:20:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 47010816 bytes 2023-12-04T14:20:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 85507072 bytes 2023-12-04T14:20:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 122405888 bytes 2023-12-04T14:20:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 203451392 bytes 2023-12-04T14:20:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 268990464 bytes 2023-12-04T14:20:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 338061312 bytes 2023-12-04T14:20:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 418463744 bytes 2023-12-04T14:20:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 419097600 bytes 2023-12-04T14:20:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 455904256 bytes 2023-12-04T14:20:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 489378816 bytes 2023-12-04T14:20:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 504151040 bytes 2023-12-04T14:20:49.096 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:20:49.094+0000 7f4bfa770640 -1 reset not still connected to 0x56230f5b8f70 2023-12-04T14:20:49.096 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:20:49.094+0000 7f4bfa770640 -1 reset not still connected to 0x562310005380 2023-12-04T14:20:49.096 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:20:49.094+0000 7f4bfa770640 -1 reset not still connected to 0x56231a33f040 2023-12-04T14:20:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 561728512 bytes 2023-12-04T14:20:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 567649280 bytes 2023-12-04T14:20:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 588310528 bytes 2023-12-04T14:20:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 670443520 bytes 2023-12-04T14:20:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 723793920 bytes 2023-12-04T14:20:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 766328832 bytes 2023-12-04T14:20:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 801453056 bytes 2023-12-04T14:20:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 832350208 bytes 2023-12-04T14:20:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 858318848 bytes 2023-12-04T14:20:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 901694464 bytes 2023-12-04T14:20:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 940707840 bytes 2023-12-04T14:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1012082688 bytes 2023-12-04T14:20:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1021042688 bytes 2023-12-04T14:20:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1100738560 bytes 2023-12-04T14:20:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1154443264 bytes 2023-12-04T14:20:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1178263552 bytes 2023-12-04T14:20:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1247467520 bytes 2023-12-04T14:20:52.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1345039360 bytes 2023-12-04T14:20:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1374630912 bytes 2023-12-04T14:20:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1403536384 bytes 2023-12-04T14:20:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1475352576 bytes 2023-12-04T14:20:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1486811136 bytes 2023-12-04T14:20:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1552206848 bytes 2023-12-04T14:20:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1595830272 bytes 2023-12-04T14:20:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1664288768 bytes 2023-12-04T14:20:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1709417472 bytes 2023-12-04T14:20:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:20:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:20:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:20:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:20:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:20:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_20 2023-12-04T14:20:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-12-04T14:20:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:20:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:20:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:20:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_20 128M 2023-12-04T14:20:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:20:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:20:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:20:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:20:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:20:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_20 2023-12-04T14:20:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:20:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:20:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:20:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-12-04T14:20:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:20:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:20:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-12-04T14:20:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:20:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-12-04T14:20:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_20': 2023-12-04T14:20:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 133a5a3d3ac9 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.133a5a3d3ac9 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:53 2023 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:20:53 2023 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:53 2023 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:20:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:20:54.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: e7fd68b0-bd04-4c3c-af90-fd5517fef987 2023-12-04T14:20:54.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:20:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:20:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_20 2023-12-04T14:20:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:20:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:20:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:20:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:20:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=6 2023-12-04T14:20:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:20:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:20:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 53421056 bytes 2023-12-04T14:20:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 111600640 bytes 2023-12-04T14:20:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 132897792 bytes 2023-12-04T14:20:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 207370240 bytes 2023-12-04T14:20:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 255120384 bytes 2023-12-04T14:20:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 335963136 bytes 2023-12-04T14:20:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 386937856 bytes 2023-12-04T14:20:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 453882880 bytes 2023-12-04T14:20:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 454499328 bytes 2023-12-04T14:20:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 494922752 bytes 2023-12-04T14:20:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 564719616 bytes 2023-12-04T14:20:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 628196352 bytes 2023-12-04T14:20:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 672659456 bytes 2023-12-04T14:20:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 748250112 bytes 2023-12-04T14:20:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 802219008 bytes 2023-12-04T14:20:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 869914624 bytes 2023-12-04T14:20:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 888568832 bytes 2023-12-04T14:20:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 929746944 bytes 2023-12-04T14:20:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 1045366784 bytes 2023-12-04T14:20:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 1117895680 bytes 2023-12-04T14:20:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 1172002816 bytes 2023-12-04T14:20:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1194272768 bytes 2023-12-04T14:20:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1244377088 bytes 2023-12-04T14:20:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1294535680 bytes 2023-12-04T14:20:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1336212480 bytes 2023-12-04T14:21:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1424791552 bytes 2023-12-04T14:21:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1447068672 bytes 2023-12-04T14:21:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1482841088 bytes 2023-12-04T14:21:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1518653440 bytes 2023-12-04T14:21:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1551008768 bytes 2023-12-04T14:21:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:21:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_21 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_21 128M 2023-12-04T14:21:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:21:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:21:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:21:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:21:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_21 2023-12-04T14:21:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:21:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:21:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:21:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-12-04T14:21:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:21:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:21:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-12-04T14:21:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:21:01.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-12-04T14:21:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_21': 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 134671cdde65 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.134671cdde65 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:00 2023 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:21:00 2023 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:00 2023 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 2a201e15-51f0-492a-8b9c-28ef7a490beb 2023-12-04T14:21:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:21:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:21:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_21 2023-12-04T14:21:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:21:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:21:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=9 2023-12-04T14:21:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:21:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:21:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 33562624 bytes 2023-12-04T14:21:01.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 62872576 bytes 2023-12-04T14:21:01.518 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7797136773461458) > weight (0.25) 2023-12-04T14:21:01.518 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-04T14:21:01.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-04T14:21:01.519 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-04T14:21:01.519 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-04T14:21:01.519 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:21:01.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi176.stderr:2023-12-04T14:21:01.518+0000 7f8981a0b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 99980) UID: 1000 2023-12-04T14:21:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 90577920 bytes 2023-12-04T14:21:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 91191296 bytes 2023-12-04T14:21:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 113495040 bytes 2023-12-04T14:21:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 137434112 bytes 2023-12-04T14:21:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 204731392 bytes 2023-12-04T14:21:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 238285824 bytes 2023-12-04T14:21:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 253627392 bytes 2023-12-04T14:21:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 286560256 bytes 2023-12-04T14:21:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 350759936 bytes 2023-12-04T14:21:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 388310016 bytes 2023-12-04T14:21:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 439138304 bytes 2023-12-04T14:21:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 484012032 bytes 2023-12-04T14:21:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 538616832 bytes 2023-12-04T14:21:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 604395520 bytes 2023-12-04T14:21:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 630301696 bytes 2023-12-04T14:21:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 672436224 bytes 2023-12-04T14:21:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 676420608 bytes 2023-12-04T14:21:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 719158272 bytes 2023-12-04T14:21:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 804032512 bytes 2023-12-04T14:21:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 831307776 bytes 2023-12-04T14:21:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 861947904 bytes 2023-12-04T14:21:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 913810432 bytes 2023-12-04T14:21:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 932131840 bytes 2023-12-04T14:21:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 963903488 bytes 2023-12-04T14:21:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1024513024 bytes 2023-12-04T14:21:05.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1075632128 bytes 2023-12-04T14:21:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1120904192 bytes 2023-12-04T14:21:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1173319680 bytes 2023-12-04T14:21:06.298 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-04T14:21:06.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-04T14:21:06.299 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-04T14:21:06.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi176.stderr:2023-12-04T14:21:06.298+0000 7f1aa731c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 99091) UID: 1000 2023-12-04T14:21:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1195340800 bytes 2023-12-04T14:21:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1262149632 bytes 2023-12-04T14:21:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1300281344 bytes 2023-12-04T14:21:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1358846976 bytes 2023-12-04T14:21:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1394680832 bytes 2023-12-04T14:21:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1433741312 bytes 2023-12-04T14:21:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1472345088 bytes 2023-12-04T14:21:07.621 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-04T14:21:07.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-04T14:21:07.622 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-04T14:21:07.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-04T14:21:07.625 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2023-12-04T14:21:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1483517952 bytes 2023-12-04T14:21:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1558415360 bytes 2023-12-04T14:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1559035904 bytes 2023-12-04T14:21:08.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1612502016 bytes 2023-12-04T14:21:08.299 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-04T14:21:08.299 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:21:08.300 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-04T14:21:08.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-04T14:21:08.300 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-04T14:21:08.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-04T14:21:08.306 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2023-12-04T14:21:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1643127808 bytes 2023-12-04T14:21:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1706174464 bytes 2023-12-04T14:21:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1760600064 bytes 2023-12-04T14:21:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1832065024 bytes 2023-12-04T14:21:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1888857088 bytes 2023-12-04T14:21:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1902795776 bytes 2023-12-04T14:21:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1933828096 bytes 2023-12-04T14:21:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 1969156096 bytes 2023-12-04T14:21:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2039412736 bytes 2023-12-04T14:21:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:21:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:21:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:21:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:21:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:21:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_22 2023-12-04T14:21:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-12-04T14:21:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:21:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:21:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:21:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_22 128M 2023-12-04T14:21:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:21:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:21:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:21:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:21:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_22 2023-12-04T14:21:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:21:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:21:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:21:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-12-04T14:21:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:21:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:21:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-12-04T14:21:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:21:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-12-04T14:21:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_22': 2023-12-04T14:21:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:21:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:21:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:21:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 135f8ef56a1 2023-12-04T14:21:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.135f8ef56a1 2023-12-04T14:21:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:21:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:21:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:21:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:21:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:10 2023 2023-12-04T14:21:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:21:10 2023 2023-12-04T14:21:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:10 2023 2023-12-04T14:21:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:21:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:21:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 059e9df9-d50d-4eb9-8306-2dcc0c369e9c 2023-12-04T14:21:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:21:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:21:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_22 2023-12-04T14:21:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:21:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:21:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=13 2023-12-04T14:21:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:21:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:21:11.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 68116480 bytes 2023-12-04T14:21:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 82471936 bytes 2023-12-04T14:21:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 132771840 bytes 2023-12-04T14:21:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 177542144 bytes 2023-12-04T14:21:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 237048832 bytes 2023-12-04T14:21:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 362163200 bytes 2023-12-04T14:21:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 424221696 bytes 2023-12-04T14:21:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 437998592 bytes 2023-12-04T14:21:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 462356480 bytes 2023-12-04T14:21:13.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 494049280 bytes 2023-12-04T14:21:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 522334208 bytes 2023-12-04T14:21:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 584951808 bytes 2023-12-04T14:21:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 606445568 bytes 2023-12-04T14:21:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 629875712 bytes 2023-12-04T14:21:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 662797312 bytes 2023-12-04T14:21:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 686026752 bytes 2023-12-04T14:21:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 750242816 bytes 2023-12-04T14:21:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 800749568 bytes 2023-12-04T14:21:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 825280512 bytes 2023-12-04T14:21:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 886029312 bytes 2023-12-04T14:21:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 893377536 bytes 2023-12-04T14:21:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 895822848 bytes 2023-12-04T14:21:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 896415744 bytes 2023-12-04T14:21:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 914379776 bytes 2023-12-04T14:21:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 917511168 bytes 2023-12-04T14:21:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 945553408 bytes 2023-12-04T14:21:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1021417472 bytes 2023-12-04T14:21:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1109026816 bytes 2023-12-04T14:21:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1150544896 bytes 2023-12-04T14:21:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1186841600 bytes 2023-12-04T14:21:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1213118464 bytes 2023-12-04T14:21:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1313604608 bytes 2023-12-04T14:21:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1424162816 bytes 2023-12-04T14:21:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1507744768 bytes 2023-12-04T14:21:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1519555584 bytes 2023-12-04T14:21:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1603566592 bytes 2023-12-04T14:21:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1693874176 bytes 2023-12-04T14:21:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1716890624 bytes 2023-12-04T14:21:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1737911296 bytes 2023-12-04T14:21:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1806460928 bytes 2023-12-04T14:21:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1826674688 bytes 2023-12-04T14:21:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1856850944 bytes 2023-12-04T14:21:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1870982144 bytes 2023-12-04T14:21:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1904606208 bytes 2023-12-04T14:21:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1959275520 bytes 2023-12-04T14:21:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1971877888 bytes 2023-12-04T14:21:19.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2017425408 bytes 2023-12-04T14:21:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2054016000 bytes 2023-12-04T14:21:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2063842304 bytes 2023-12-04T14:21:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2105469952 bytes 2023-12-04T14:21:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2172398592 bytes 2023-12-04T14:21:20.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2260035584 bytes 2023-12-04T14:21:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2284849152 bytes 2023-12-04T14:21:20.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2331120640 bytes 2023-12-04T14:21:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2419781632 bytes 2023-12-04T14:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2453672960 bytes 2023-12-04T14:21:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2473849856 bytes 2023-12-04T14:21:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2541418496 bytes 2023-12-04T14:21:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2561655808 bytes 2023-12-04T14:21:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2582910976 bytes 2023-12-04T14:21:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2589844480 bytes 2023-12-04T14:21:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2591984640 bytes 2023-12-04T14:21:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2625075200 bytes 2023-12-04T14:21:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2691574784 bytes 2023-12-04T14:21:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2748296192 bytes 2023-12-04T14:21:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2864304128 bytes 2023-12-04T14:21:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2911505408 bytes 2023-12-04T14:21:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2951397376 bytes 2023-12-04T14:21:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2982577152 bytes 2023-12-04T14:21:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 3042605056 bytes 2023-12-04T14:21:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 3059919872 bytes 2023-12-04T14:21:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_23 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_23 128M 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_23 2023-12-04T14:21:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:21:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:21:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:21:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-12-04T14:21:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:21:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:21:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-12-04T14:21:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:21:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-12-04T14:21:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_23': 2023-12-04T14:21:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:21:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:21:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:21:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 136bb9028911 2023-12-04T14:21:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.136bb9028911 2023-12-04T14:21:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:21:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:21:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:21:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:21:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:24 2023 2023-12-04T14:21:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:21:24 2023 2023-12-04T14:21:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:24 2023 2023-12-04T14:21:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:21:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:21:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: e304857c-103b-447a-ae76-3be95494c2ac 2023-12-04T14:21:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:21:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:21:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_23 2023-12-04T14:21:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:21:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:21:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=13 2023-12-04T14:21:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:21:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:21:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 60296192 bytes 2023-12-04T14:21:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 71531520 bytes 2023-12-04T14:21:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 115251200 bytes 2023-12-04T14:21:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 189581312 bytes 2023-12-04T14:21:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 252316672 bytes 2023-12-04T14:21:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 302794752 bytes 2023-12-04T14:21:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 308825088 bytes 2023-12-04T14:21:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 406755328 bytes 2023-12-04T14:21:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 470468608 bytes 2023-12-04T14:21:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 471097344 bytes 2023-12-04T14:21:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 528577536 bytes 2023-12-04T14:21:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 594648064 bytes 2023-12-04T14:21:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 611580928 bytes 2023-12-04T14:21:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 614528000 bytes 2023-12-04T14:21:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 652575744 bytes 2023-12-04T14:21:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 690136064 bytes 2023-12-04T14:21:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 722451456 bytes 2023-12-04T14:21:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 752173056 bytes 2023-12-04T14:21:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 845845504 bytes 2023-12-04T14:21:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 866593792 bytes 2023-12-04T14:21:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 923694080 bytes 2023-12-04T14:21:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 996900864 bytes 2023-12-04T14:21:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1110372352 bytes 2023-12-04T14:21:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1138670592 bytes 2023-12-04T14:21:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1149365248 bytes 2023-12-04T14:21:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1225757696 bytes 2023-12-04T14:21:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1265153024 bytes 2023-12-04T14:21:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1295967232 bytes 2023-12-04T14:21:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1377760256 bytes 2023-12-04T14:21:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1416289280 bytes 2023-12-04T14:21:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1457709056 bytes 2023-12-04T14:21:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1498302464 bytes 2023-12-04T14:21:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1532162048 bytes 2023-12-04T14:21:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1553959936 bytes 2023-12-04T14:21:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1600083968 bytes 2023-12-04T14:21:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1624815616 bytes 2023-12-04T14:21:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1648790528 bytes 2023-12-04T14:21:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1719455744 bytes 2023-12-04T14:21:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1741569024 bytes 2023-12-04T14:21:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1760115712 bytes 2023-12-04T14:21:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1820531712 bytes 2023-12-04T14:21:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1880845312 bytes 2023-12-04T14:21:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1929621504 bytes 2023-12-04T14:21:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 2019749888 bytes 2023-12-04T14:21:33.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 2070223872 bytes 2023-12-04T14:21:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 2090710016 bytes 2023-12-04T14:21:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2134553600 bytes 2023-12-04T14:21:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2181125120 bytes 2023-12-04T14:21:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2239942656 bytes 2023-12-04T14:21:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2276779008 bytes 2023-12-04T14:21:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2297491456 bytes 2023-12-04T14:21:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2340710400 bytes 2023-12-04T14:21:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2363168768 bytes 2023-12-04T14:21:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2418200576 bytes 2023-12-04T14:21:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2419397632 bytes 2023-12-04T14:21:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2556208128 bytes 2023-12-04T14:21:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2620852224 bytes 2023-12-04T14:21:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2668032000 bytes 2023-12-04T14:21:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2692653056 bytes 2023-12-04T14:21:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2769602560 bytes 2023-12-04T14:21:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2810227712 bytes 2023-12-04T14:21:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2862065664 bytes 2023-12-04T14:21:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2915405824 bytes 2023-12-04T14:21:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2958743552 bytes 2023-12-04T14:21:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2959376384 bytes 2023-12-04T14:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2974737408 bytes 2023-12-04T14:21:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 3000896512 bytes 2023-12-04T14:21:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:21:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:21:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:21:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:21:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:21:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_24 2023-12-04T14:21:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-12-04T14:21:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:21:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:21:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:21:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_24 128M 2023-12-04T14:21:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:21:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:21:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:21:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:21:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_24 2023-12-04T14:21:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:21:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:21:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:21:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-12-04T14:21:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:21:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:21:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-12-04T14:21:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:21:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-12-04T14:21:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_24': 2023-12-04T14:21:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:21:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:21:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:21:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 13776f99bc51 2023-12-04T14:21:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.13776f99bc51 2023-12-04T14:21:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:21:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:21:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:21:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:21:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:37 2023 2023-12-04T14:21:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:21:37 2023 2023-12-04T14:21:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:37 2023 2023-12-04T14:21:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:21:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:21:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: eb8df2f9-471f-4269-a2e3-70777410572c 2023-12-04T14:21:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:21:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:21:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_24 2023-12-04T14:21:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:21:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:21:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=13 2023-12-04T14:21:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:21:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:21:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 56965120 bytes 2023-12-04T14:21:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 57570304 bytes 2023-12-04T14:21:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 95084544 bytes 2023-12-04T14:21:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 140188672 bytes 2023-12-04T14:21:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 176181248 bytes 2023-12-04T14:21:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 245965824 bytes 2023-12-04T14:21:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 267074560 bytes 2023-12-04T14:21:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 363732992 bytes 2023-12-04T14:21:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 465610752 bytes 2023-12-04T14:21:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 520455168 bytes 2023-12-04T14:21:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 541675520 bytes 2023-12-04T14:21:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 566806528 bytes 2023-12-04T14:21:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 655242240 bytes 2023-12-04T14:21:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 751090688 bytes 2023-12-04T14:21:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 807142400 bytes 2023-12-04T14:21:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 813532160 bytes 2023-12-04T14:21:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 856251392 bytes 2023-12-04T14:21:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 931106816 bytes 2023-12-04T14:21:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 963905536 bytes 2023-12-04T14:21:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 1006551040 bytes 2023-12-04T14:21:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 1039915008 bytes 2023-12-04T14:21:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1101504512 bytes 2023-12-04T14:21:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1127880704 bytes 2023-12-04T14:21:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1128500224 bytes 2023-12-04T14:21:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1229000704 bytes 2023-12-04T14:21:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1238323200 bytes 2023-12-04T14:21:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1267481600 bytes 2023-12-04T14:21:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1306591232 bytes 2023-12-04T14:21:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1351041024 bytes 2023-12-04T14:21:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1384282112 bytes 2023-12-04T14:21:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1426439168 bytes 2023-12-04T14:21:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1478894592 bytes 2023-12-04T14:21:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1504165888 bytes 2023-12-04T14:21:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1506569216 bytes 2023-12-04T14:21:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1553823744 bytes 2023-12-04T14:21:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1570043904 bytes 2023-12-04T14:21:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1570656256 bytes 2023-12-04T14:21:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1596091392 bytes 2023-12-04T14:21:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1644209152 bytes 2023-12-04T14:21:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1710188544 bytes 2023-12-04T14:21:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1771123712 bytes 2023-12-04T14:21:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1808754688 bytes 2023-12-04T14:21:45.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1830005760 bytes 2023-12-04T14:21:46.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1868759040 bytes 2023-12-04T14:21:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1925426176 bytes 2023-12-04T14:21:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1971399680 bytes 2023-12-04T14:21:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1982016512 bytes 2023-12-04T14:21:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2047273984 bytes 2023-12-04T14:21:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2060555264 bytes 2023-12-04T14:21:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2107261952 bytes 2023-12-04T14:21:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2172526592 bytes 2023-12-04T14:21:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2195449856 bytes 2023-12-04T14:21:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2230125568 bytes 2023-12-04T14:21:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2299051008 bytes 2023-12-04T14:21:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2369729536 bytes 2023-12-04T14:21:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2392839168 bytes 2023-12-04T14:21:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2444566528 bytes 2023-12-04T14:21:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2504477696 bytes 2023-12-04T14:21:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2563690496 bytes 2023-12-04T14:21:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2637869056 bytes 2023-12-04T14:21:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2676400128 bytes 2023-12-04T14:21:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2705636352 bytes 2023-12-04T14:21:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2714713088 bytes 2023-12-04T14:21:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2730204160 bytes 2023-12-04T14:21:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2763895808 bytes 2023-12-04T14:21:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2776717312 bytes 2023-12-04T14:21:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2861775872 bytes 2023-12-04T14:21:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2874900480 bytes 2023-12-04T14:21:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2885564416 bytes 2023-12-04T14:21:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 2926673920 bytes 2023-12-04T14:21:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:21:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_25 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_25 128M 2023-12-04T14:21:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:21:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:21:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:21:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:21:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:21:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_25 2023-12-04T14:21:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:21:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:21:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:21:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-12-04T14:21:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:21:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:21:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-12-04T14:21:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:21:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-12-04T14:21:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_25': 2023-12-04T14:21:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:21:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:21:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:21:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 138320899233 2023-12-04T14:21:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.138320899233 2023-12-04T14:21:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:21:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:21:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:21:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:21:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:51 2023 2023-12-04T14:21:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:21:51 2023 2023-12-04T14:21:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:51 2023 2023-12-04T14:21:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:21:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:21:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 44871cb2-b60d-4e0e-8e66-9d9bdcbae254 2023-12-04T14:21:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:21:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:21:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_25 2023-12-04T14:21:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:21:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:21:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:21:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:21:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=13 2023-12-04T14:21:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:21:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:21:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 22874112 bytes 2023-12-04T14:21:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 45705216 bytes 2023-12-04T14:21:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 83419136 bytes 2023-12-04T14:21:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 101259264 bytes 2023-12-04T14:21:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 114623488 bytes 2023-12-04T14:21:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 131003392 bytes 2023-12-04T14:21:53.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 156326912 bytes 2023-12-04T14:21:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 169825280 bytes 2023-12-04T14:21:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 213639168 bytes 2023-12-04T14:21:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 289826816 bytes 2023-12-04T14:21:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 325404672 bytes 2023-12-04T14:21:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 405206016 bytes 2023-12-04T14:21:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 459473920 bytes 2023-12-04T14:21:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 490269696 bytes 2023-12-04T14:21:54.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 518620160 bytes 2023-12-04T14:21:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 562343936 bytes 2023-12-04T14:21:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 594456576 bytes 2023-12-04T14:21:55.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 635771904 bytes 2023-12-04T14:21:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 679790592 bytes 2023-12-04T14:21:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 733446144 bytes 2023-12-04T14:21:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 734086144 bytes 2023-12-04T14:21:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 762003456 bytes 2023-12-04T14:21:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 840784896 bytes 2023-12-04T14:21:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 931197952 bytes 2023-12-04T14:21:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 970673152 bytes 2023-12-04T14:21:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 989834240 bytes 2023-12-04T14:21:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1045106688 bytes 2023-12-04T14:21:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1072366592 bytes 2023-12-04T14:21:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1125899264 bytes 2023-12-04T14:21:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1151496192 bytes 2023-12-04T14:21:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1230264320 bytes 2023-12-04T14:21:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1337320448 bytes 2023-12-04T14:21:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1341491200 bytes 2023-12-04T14:21:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1407060992 bytes 2023-12-04T14:21:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1414345728 bytes 2023-12-04T14:21:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1450964992 bytes 2023-12-04T14:21:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1502511104 bytes 2023-12-04T14:21:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1526761472 bytes 2023-12-04T14:21:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1538479104 bytes 2023-12-04T14:21:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1539105792 bytes 2023-12-04T14:21:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1572751360 bytes 2023-12-04T14:21:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1646103552 bytes 2023-12-04T14:21:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1697240064 bytes 2023-12-04T14:22:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1763007488 bytes 2023-12-04T14:22:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1784196096 bytes 2023-12-04T14:22:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1803784192 bytes 2023-12-04T14:22:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1884012544 bytes 2023-12-04T14:22:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1914386432 bytes 2023-12-04T14:22:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 1934885888 bytes 2023-12-04T14:22:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 1953438720 bytes 2023-12-04T14:22:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2031219712 bytes 2023-12-04T14:22:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2081605632 bytes 2023-12-04T14:22:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2120565760 bytes 2023-12-04T14:22:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2146995200 bytes 2023-12-04T14:22:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2179042304 bytes 2023-12-04T14:22:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2179656704 bytes 2023-12-04T14:22:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2214146048 bytes 2023-12-04T14:22:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2243422208 bytes 2023-12-04T14:22:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2264293376 bytes 2023-12-04T14:22:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2346948608 bytes 2023-12-04T14:22:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2385130496 bytes 2023-12-04T14:22:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2433039360 bytes 2023-12-04T14:22:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2502564864 bytes 2023-12-04T14:22:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2503179264 bytes 2023-12-04T14:22:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2540379136 bytes 2023-12-04T14:22:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2588887040 bytes 2023-12-04T14:22:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2633226240 bytes 2023-12-04T14:22:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2654011392 bytes 2023-12-04T14:22:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2742754304 bytes 2023-12-04T14:22:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 2782698496 bytes 2023-12-04T14:22:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 2870565888 bytes 2023-12-04T14:22:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 2907822080 bytes 2023-12-04T14:22:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:22:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:22:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_26 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_26 128M 2023-12-04T14:22:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:22:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:22:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:22:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:22:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_26 2023-12-04T14:22:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:22:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:22:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-12-04T14:22:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:22:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-12-04T14:22:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:22:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-12-04T14:22:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_26': 2023-12-04T14:22:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:22:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:22:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:22:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 138f9234722 2023-12-04T14:22:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.138f9234722 2023-12-04T14:22:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:22:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:22:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:22:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:22:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:05 2023 2023-12-04T14:22:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:05 2023 2023-12-04T14:22:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:05 2023 2023-12-04T14:22:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:22:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:22:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: edd4af1f-3c2b-4169-adc5-5d59c2cc5a61 2023-12-04T14:22:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:22:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:22:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_26 2023-12-04T14:22:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:22:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:22:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=5 2023-12-04T14:22:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:22:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:22:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 73873408 bytes 2023-12-04T14:22:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 108028928 bytes 2023-12-04T14:22:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 185180160 bytes 2023-12-04T14:22:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 251419648 bytes 2023-12-04T14:22:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 284039168 bytes 2023-12-04T14:22:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 329633792 bytes 2023-12-04T14:22:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 362092544 bytes 2023-12-04T14:22:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 404044800 bytes 2023-12-04T14:22:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 444141568 bytes 2023-12-04T14:22:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 462400512 bytes 2023-12-04T14:22:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 514467840 bytes 2023-12-04T14:22:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 573376512 bytes 2023-12-04T14:22:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 695149568 bytes 2023-12-04T14:22:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 774169600 bytes 2023-12-04T14:22:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 801186816 bytes 2023-12-04T14:22:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 832062464 bytes 2023-12-04T14:22:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 863186944 bytes 2023-12-04T14:22:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 893552640 bytes 2023-12-04T14:22:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 926944256 bytes 2023-12-04T14:22:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 957324288 bytes 2023-12-04T14:22:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 1007291392 bytes 2023-12-04T14:22:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1054456832 bytes 2023-12-04T14:22:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1113818112 bytes 2023-12-04T14:22:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1127463936 bytes 2023-12-04T14:22:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1173379072 bytes 2023-12-04T14:22:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:22:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:22:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:22:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:22:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_27 2023-12-04T14:22:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_27 128M 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:22:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:22:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:22:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:22:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_27 2023-12-04T14:22:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:22:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:22:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-12-04T14:22:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:22:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-12-04T14:22:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:22:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-12-04T14:22:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_27': 2023-12-04T14:22:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:22:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:22:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:22:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 139b43c83599 2023-12-04T14:22:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.139b43c83599 2023-12-04T14:22:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:22:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:22:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:22:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:22:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:10 2023 2023-12-04T14:22:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:10 2023 2023-12-04T14:22:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:10 2023 2023-12-04T14:22:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:22:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:22:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: e393e7c2-3dbe-4e5b-9517-da1c79d5fe47 2023-12-04T14:22:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:22:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:22:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_27 2023-12-04T14:22:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:22:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:22:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=11 2023-12-04T14:22:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:22:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:22:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 20020224 bytes 2023-12-04T14:22:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 47242240 bytes 2023-12-04T14:22:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 162760704 bytes 2023-12-04T14:22:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 186790912 bytes 2023-12-04T14:22:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 237308928 bytes 2023-12-04T14:22:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 279336960 bytes 2023-12-04T14:22:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 299180032 bytes 2023-12-04T14:22:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 307917824 bytes 2023-12-04T14:22:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 370688000 bytes 2023-12-04T14:22:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 376973312 bytes 2023-12-04T14:22:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 407934976 bytes 2023-12-04T14:22:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 450329600 bytes 2023-12-04T14:22:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 473486336 bytes 2023-12-04T14:22:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 488967168 bytes 2023-12-04T14:22:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 531023872 bytes 2023-12-04T14:22:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 587764736 bytes 2023-12-04T14:22:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 669238272 bytes 2023-12-04T14:22:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 717435904 bytes 2023-12-04T14:22:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 771300352 bytes 2023-12-04T14:22:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 811584512 bytes 2023-12-04T14:22:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 832820224 bytes 2023-12-04T14:22:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 882433024 bytes 2023-12-04T14:22:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 918792192 bytes 2023-12-04T14:22:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 936632320 bytes 2023-12-04T14:22:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 977968128 bytes 2023-12-04T14:22:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1006376960 bytes 2023-12-04T14:22:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1090400256 bytes 2023-12-04T14:22:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1157891072 bytes 2023-12-04T14:22:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1227481088 bytes 2023-12-04T14:22:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1260072960 bytes 2023-12-04T14:22:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1260682240 bytes 2023-12-04T14:22:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1345573888 bytes 2023-12-04T14:22:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1412504576 bytes 2023-12-04T14:22:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1462408192 bytes 2023-12-04T14:22:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1561671680 bytes 2023-12-04T14:22:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1624359936 bytes 2023-12-04T14:22:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1664859136 bytes 2023-12-04T14:22:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1703639040 bytes 2023-12-04T14:22:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1749935104 bytes 2023-12-04T14:22:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1829459968 bytes 2023-12-04T14:22:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1928777728 bytes 2023-12-04T14:22:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1981179904 bytes 2023-12-04T14:22:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 2056807424 bytes 2023-12-04T14:22:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 2083816448 bytes 2023-12-04T14:22:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 2140528640 bytes 2023-12-04T14:22:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 2196111360 bytes 2023-12-04T14:22:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2226197504 bytes 2023-12-04T14:22:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2295849984 bytes 2023-12-04T14:22:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2371895296 bytes 2023-12-04T14:22:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2419197952 bytes 2023-12-04T14:22:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2492996608 bytes 2023-12-04T14:22:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2512979968 bytes 2023-12-04T14:22:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2603955200 bytes 2023-12-04T14:22:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2683183104 bytes 2023-12-04T14:22:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2787559424 bytes 2023-12-04T14:22:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:22:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:22:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:22:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:22:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_28 2023-12-04T14:22:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-12-04T14:22:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_28 128M 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:22:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:22:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:22:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:22:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_28 2023-12-04T14:22:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:22:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:22:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-12-04T14:22:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:22:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-12-04T14:22:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:22:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-12-04T14:22:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_28': 2023-12-04T14:22:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:22:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:22:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 13a73761e5af 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.13a73761e5af 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:22 2023 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:22 2023 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:22 2023 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: ad2b0660-9c28-48f0-8b26-08f4860ad09e 2023-12-04T14:22:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:22:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:22:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_28 2023-12-04T14:22:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:22:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:22:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=14 2023-12-04T14:22:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:22:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:22:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 30093312 bytes 2023-12-04T14:22:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 60163072 bytes 2023-12-04T14:22:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 120862720 bytes 2023-12-04T14:22:24.009 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 216696832 bytes 2023-12-04T14:22:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 258492416 bytes 2023-12-04T14:22:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 311201792 bytes 2023-12-04T14:22:24.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 349836288 bytes 2023-12-04T14:22:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 409162752 bytes 2023-12-04T14:22:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 447439872 bytes 2023-12-04T14:22:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 475323392 bytes 2023-12-04T14:22:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 487312384 bytes 2023-12-04T14:22:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 506409984 bytes 2023-12-04T14:22:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 570426368 bytes 2023-12-04T14:22:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 608886784 bytes 2023-12-04T14:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 677609472 bytes 2023-12-04T14:22:26.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 769708032 bytes 2023-12-04T14:22:26.426 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:22:26.425+0000 7f8e4ae66640 -1 reset not still connected to 0x55b1a3110820 2023-12-04T14:22:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 795380736 bytes 2023-12-04T14:22:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 850748416 bytes 2023-12-04T14:22:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 911081472 bytes 2023-12-04T14:22:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 933057536 bytes 2023-12-04T14:22:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 936408064 bytes 2023-12-04T14:22:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 941998080 bytes 2023-12-04T14:22:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 968061952 bytes 2023-12-04T14:22:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1043230720 bytes 2023-12-04T14:22:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1043819520 bytes 2023-12-04T14:22:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1102387200 bytes 2023-12-04T14:22:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1103002624 bytes 2023-12-04T14:22:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1116641280 bytes 2023-12-04T14:22:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1159522304 bytes 2023-12-04T14:22:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1178726400 bytes 2023-12-04T14:22:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1179340800 bytes 2023-12-04T14:22:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1302434816 bytes 2023-12-04T14:22:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1359259648 bytes 2023-12-04T14:22:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1360635904 bytes 2023-12-04T14:22:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1368238080 bytes 2023-12-04T14:22:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1432570880 bytes 2023-12-04T14:22:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1480981504 bytes 2023-12-04T14:22:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1539576832 bytes 2023-12-04T14:22:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1567579136 bytes 2023-12-04T14:22:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1620032512 bytes 2023-12-04T14:22:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1672208384 bytes 2023-12-04T14:22:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1748170752 bytes 2023-12-04T14:22:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1808339968 bytes 2023-12-04T14:22:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1883006976 bytes 2023-12-04T14:22:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1928774656 bytes 2023-12-04T14:22:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1997340672 bytes 2023-12-04T14:22:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2050838528 bytes 2023-12-04T14:22:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2076041216 bytes 2023-12-04T14:22:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2180384768 bytes 2023-12-04T14:22:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2203302912 bytes 2023-12-04T14:22:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2246506496 bytes 2023-12-04T14:22:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2317723648 bytes 2023-12-04T14:22:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2330730496 bytes 2023-12-04T14:22:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2351035392 bytes 2023-12-04T14:22:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2403580928 bytes 2023-12-04T14:22:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2449989632 bytes 2023-12-04T14:22:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2454370304 bytes 2023-12-04T14:22:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2535866368 bytes 2023-12-04T14:22:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2567153664 bytes 2023-12-04T14:22:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2578284544 bytes 2023-12-04T14:22:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2644648960 bytes 2023-12-04T14:22:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2674068480 bytes 2023-12-04T14:22:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2754972672 bytes 2023-12-04T14:22:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2843560960 bytes 2023-12-04T14:22:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2893327360 bytes 2023-12-04T14:22:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2956033024 bytes 2023-12-04T14:22:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2990178304 bytes 2023-12-04T14:22:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 3010108416 bytes 2023-12-04T14:22:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 3017114624 bytes 2023-12-04T14:22:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 3116956672 bytes 2023-12-04T14:22:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 3117550592 bytes 2023-12-04T14:22:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 3173088256 bytes 2023-12-04T14:22:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7300 IOs, 3217276928 bytes 2023-12-04T14:22:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:22:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:22:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:22:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:22:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_29 2023-12-04T14:22:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-12-04T14:22:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_29 128M 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:22:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:22:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:22:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_29 2023-12-04T14:22:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:22:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:22:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-12-04T14:22:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:22:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-12-04T14:22:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:22:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-12-04T14:22:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_29': 2023-12-04T14:22:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:22:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:22:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 13b3c97ad9fb 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.13b3c97ad9fb 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:37 2023 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:37 2023 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:37 2023 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 040b1033-d414-4ab1-bb84-83c330f7ff65 2023-12-04T14:22:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:22:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:22:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_29 2023-12-04T14:22:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:22:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:22:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=5 2023-12-04T14:22:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:22:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:22:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 40585216 bytes 2023-12-04T14:22:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 72492032 bytes 2023-12-04T14:22:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 112644096 bytes 2023-12-04T14:22:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 152638464 bytes 2023-12-04T14:22:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 179629056 bytes 2023-12-04T14:22:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 260273152 bytes 2023-12-04T14:22:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 301328384 bytes 2023-12-04T14:22:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 379858944 bytes 2023-12-04T14:22:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 440603648 bytes 2023-12-04T14:22:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 488206336 bytes 2023-12-04T14:22:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 581096448 bytes 2023-12-04T14:22:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 621029376 bytes 2023-12-04T14:22:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 661069824 bytes 2023-12-04T14:22:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 684610560 bytes 2023-12-04T14:22:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 726770688 bytes 2023-12-04T14:22:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 782574592 bytes 2023-12-04T14:22:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 818830336 bytes 2023-12-04T14:22:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 864230400 bytes 2023-12-04T14:22:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 929067008 bytes 2023-12-04T14:22:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 1016991744 bytes 2023-12-04T14:22:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 1067675648 bytes 2023-12-04T14:22:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1134717952 bytes 2023-12-04T14:22:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1162599424 bytes 2023-12-04T14:22:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1217477632 bytes 2023-12-04T14:22:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:22:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:22:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:22:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:22:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_30 2023-12-04T14:22:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-12-04T14:22:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:22:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:22:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_30 128M 2023-12-04T14:22:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:22:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:22:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:22:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:22:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_30 2023-12-04T14:22:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:22:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:22:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-12-04T14:22:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:22:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-12-04T14:22:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:22:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-12-04T14:22:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_30': 2023-12-04T14:22:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:22:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:22:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:22:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 13bfa7f25127 2023-12-04T14:22:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.13bfa7f25127 2023-12-04T14:22:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:22:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:22:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:22:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:22:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:43 2023 2023-12-04T14:22:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:43 2023 2023-12-04T14:22:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:43 2023 2023-12-04T14:22:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:22:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:22:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 233226c3-03eb-48bb-bc7e-fe8f5e6be5a9 2023-12-04T14:22:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:22:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:22:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_30 2023-12-04T14:22:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:22:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:22:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=8 2023-12-04T14:22:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:22:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:22:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 51588096 bytes 2023-12-04T14:22:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 100024320 bytes 2023-12-04T14:22:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 176531456 bytes 2023-12-04T14:22:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 201839616 bytes 2023-12-04T14:22:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 228011008 bytes 2023-12-04T14:22:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 264531968 bytes 2023-12-04T14:22:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 295875584 bytes 2023-12-04T14:22:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 313458688 bytes 2023-12-04T14:22:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 354411520 bytes 2023-12-04T14:22:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 383451136 bytes 2023-12-04T14:22:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 401705984 bytes 2023-12-04T14:22:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 472004608 bytes 2023-12-04T14:22:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 540491776 bytes 2023-12-04T14:22:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 557208576 bytes 2023-12-04T14:22:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 652053504 bytes 2023-12-04T14:22:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 725566464 bytes 2023-12-04T14:22:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 745247744 bytes 2023-12-04T14:22:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 808487936 bytes 2023-12-04T14:22:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 853239808 bytes 2023-12-04T14:22:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 927025152 bytes 2023-12-04T14:22:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 957970432 bytes 2023-12-04T14:22:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1051969536 bytes 2023-12-04T14:22:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1081698304 bytes 2023-12-04T14:22:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1093174272 bytes 2023-12-04T14:22:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1148663808 bytes 2023-12-04T14:22:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1182456832 bytes 2023-12-04T14:22:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1203737600 bytes 2023-12-04T14:22:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1244090368 bytes 2023-12-04T14:22:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1246578688 bytes 2023-12-04T14:22:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1288138752 bytes 2023-12-04T14:22:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1356300288 bytes 2023-12-04T14:22:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1384619008 bytes 2023-12-04T14:22:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1410125824 bytes 2023-12-04T14:22:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1446025216 bytes 2023-12-04T14:22:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1495696384 bytes 2023-12-04T14:22:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1584005120 bytes 2023-12-04T14:22:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1670509568 bytes 2023-12-04T14:22:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1681765376 bytes 2023-12-04T14:22:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1693992960 bytes 2023-12-04T14:22:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1724216320 bytes 2023-12-04T14:22:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1742953472 bytes 2023-12-04T14:22:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1813244928 bytes 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_31 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:22:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_31 128M 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_31 2023-12-04T14:22:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:22:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:22:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-12-04T14:22:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:22:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-12-04T14:22:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:22:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-12-04T14:22:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_31': 2023-12-04T14:22:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:22:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:22:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:22:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 13cb5de373b7 2023-12-04T14:22:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.13cb5de373b7 2023-12-04T14:22:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:22:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:22:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:22:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:22:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:52 2023 2023-12-04T14:22:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:52 2023 2023-12-04T14:22:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:52 2023 2023-12-04T14:22:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:22:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:22:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 55aa29aa-3f60-443f-a765-b4d1a2257a41 2023-12-04T14:22:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:22:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:22:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_31 2023-12-04T14:22:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:22:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:22:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=5 2023-12-04T14:22:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:22:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:22:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 30983168 bytes 2023-12-04T14:22:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 69097472 bytes 2023-12-04T14:22:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 126493696 bytes 2023-12-04T14:22:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 213394432 bytes 2023-12-04T14:22:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 224081920 bytes 2023-12-04T14:22:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 288496640 bytes 2023-12-04T14:22:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 294546432 bytes 2023-12-04T14:22:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 313828352 bytes 2023-12-04T14:22:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 383416320 bytes 2023-12-04T14:22:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 432319488 bytes 2023-12-04T14:22:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 475424768 bytes 2023-12-04T14:22:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 549306368 bytes 2023-12-04T14:22:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 549933056 bytes 2023-12-04T14:22:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 575188992 bytes 2023-12-04T14:22:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 608990208 bytes 2023-12-04T14:22:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 657281024 bytes 2023-12-04T14:22:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 739904512 bytes 2023-12-04T14:22:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 761759744 bytes 2023-12-04T14:22:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 829218816 bytes 2023-12-04T14:22:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 829812736 bytes 2023-12-04T14:22:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 839459840 bytes 2023-12-04T14:22:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 851077120 bytes 2023-12-04T14:22:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 900127744 bytes 2023-12-04T14:22:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 918745088 bytes 2023-12-04T14:22:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 956236800 bytes 2023-12-04T14:22:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 957705216 bytes 2023-12-04T14:22:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 981656576 bytes 2023-12-04T14:22:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1025156096 bytes 2023-12-04T14:22:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_32 2023-12-04T14:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-12-04T14:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_32 128M 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:22:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:22:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_32 2023-12-04T14:22:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:22:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:22:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:22:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-12-04T14:22:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:22:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:22:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-12-04T14:22:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:22:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-12-04T14:22:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_32': 2023-12-04T14:22:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:22:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:22:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:22:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 13d7e785a495 2023-12-04T14:22:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.13d7e785a495 2023-12-04T14:22:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:22:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:22:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:22:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:22:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:58 2023 2023-12-04T14:22:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:58 2023 2023-12-04T14:22:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:58 2023 2023-12-04T14:22:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:22:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:22:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: da0438b0-0666-4c5d-bca2-497b527c3c8a 2023-12-04T14:22:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:22:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:22:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_32 2023-12-04T14:22:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:22:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:22:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:22:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:22:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=9 2023-12-04T14:22:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:22:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:22:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 23164928 bytes 2023-12-04T14:22:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 58595328 bytes 2023-12-04T14:22:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 101284864 bytes 2023-12-04T14:23:00.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 160772096 bytes 2023-12-04T14:23:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 200879104 bytes 2023-12-04T14:23:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 220809216 bytes 2023-12-04T14:23:00.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 285297664 bytes 2023-12-04T14:23:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 348854272 bytes 2023-12-04T14:23:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 349464576 bytes 2023-12-04T14:23:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 416324608 bytes 2023-12-04T14:23:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 475857920 bytes 2023-12-04T14:23:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 549016576 bytes 2023-12-04T14:23:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 573840384 bytes 2023-12-04T14:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 618900480 bytes 2023-12-04T14:23:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 634423296 bytes 2023-12-04T14:23:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 694772736 bytes 2023-12-04T14:23:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 695373824 bytes 2023-12-04T14:23:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 735273984 bytes 2023-12-04T14:23:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 808972288 bytes 2023-12-04T14:23:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 832872448 bytes 2023-12-04T14:23:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 890548224 bytes 2023-12-04T14:23:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 891173888 bytes 2023-12-04T14:23:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 933178368 bytes 2023-12-04T14:23:03.626 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3683335533329396) > weight (0.25) 2023-12-04T14:23:03.626 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6526917251403062) > weight (0.25) 2023-12-04T14:23:03.626 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4106777179633829) > weight (0.25) 2023-12-04T14:23:03.626 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8290777353485829) > weight (0.25) 2023-12-04T14:23:03.626 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2023-12-04T14:23:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 960290816 bytes 2023-12-04T14:23:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 973292544 bytes 2023-12-04T14:23:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1016706048 bytes 2023-12-04T14:23:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1032136704 bytes 2023-12-04T14:23:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1073449984 bytes 2023-12-04T14:23:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1151291392 bytes 2023-12-04T14:23:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1174580224 bytes 2023-12-04T14:23:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1235251200 bytes 2023-12-04T14:23:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1258775552 bytes 2023-12-04T14:23:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1272392704 bytes 2023-12-04T14:23:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1342351360 bytes 2023-12-04T14:23:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1398148096 bytes 2023-12-04T14:23:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1423069184 bytes 2023-12-04T14:23:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1459101696 bytes 2023-12-04T14:23:06.307 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6167000071419235) > weight (0.25) 2023-12-04T14:23:06.307 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.34890615671446956) > weight (0.25) 2023-12-04T14:23:06.307 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9184909015695365) > weight (0.25) 2023-12-04T14:23:06.307 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8721974150478845) > weight (0.25) 2023-12-04T14:23:06.307 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2023-12-04T14:23:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1524917248 bytes 2023-12-04T14:23:06.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1563456512 bytes 2023-12-04T14:23:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1603461120 bytes 2023-12-04T14:23:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1653149696 bytes 2023-12-04T14:23:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1719151616 bytes 2023-12-04T14:23:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1738531840 bytes 2023-12-04T14:23:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1801042944 bytes 2023-12-04T14:23:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1830793216 bytes 2023-12-04T14:23:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1903726592 bytes 2023-12-04T14:23:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1944780800 bytes 2023-12-04T14:23:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:23:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:23:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:23:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:23:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:23:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_33 2023-12-04T14:23:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-12-04T14:23:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:23:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:23:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:23:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_33 128M 2023-12-04T14:23:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:23:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:23:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:23:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:23:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_33 2023-12-04T14:23:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:23:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:23:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:23:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-12-04T14:23:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:23:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:23:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-12-04T14:23:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:23:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-12-04T14:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_33': 2023-12-04T14:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 13e3f63987eb 2023-12-04T14:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.13e3f63987eb 2023-12-04T14:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:08 2023 2023-12-04T14:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:23:08 2023 2023-12-04T14:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:08 2023 2023-12-04T14:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 709187ee-c3f5-47e9-a97c-a08560690fe3 2023-12-04T14:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:23:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:23:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_33 2023-12-04T14:23:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:23:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:23:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=13 2023-12-04T14:23:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:23:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:23:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 1359872 bytes 2023-12-04T14:23:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 67368960 bytes 2023-12-04T14:23:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 105920512 bytes 2023-12-04T14:23:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 138060800 bytes 2023-12-04T14:23:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 187130880 bytes 2023-12-04T14:23:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 265500672 bytes 2023-12-04T14:23:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 308845568 bytes 2023-12-04T14:23:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 406559744 bytes 2023-12-04T14:23:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 480201728 bytes 2023-12-04T14:23:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 548026368 bytes 2023-12-04T14:23:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 552554496 bytes 2023-12-04T14:23:11.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 568952832 bytes 2023-12-04T14:23:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 602398720 bytes 2023-12-04T14:23:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 645038080 bytes 2023-12-04T14:23:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 759062528 bytes 2023-12-04T14:23:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 780910592 bytes 2023-12-04T14:23:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 810214400 bytes 2023-12-04T14:23:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 873376768 bytes 2023-12-04T14:23:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 900187136 bytes 2023-12-04T14:23:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 938310656 bytes 2023-12-04T14:23:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 955746304 bytes 2023-12-04T14:23:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1003749376 bytes 2023-12-04T14:23:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1004368896 bytes 2023-12-04T14:23:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1053974528 bytes 2023-12-04T14:23:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1082603520 bytes 2023-12-04T14:23:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1151448064 bytes 2023-12-04T14:23:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1207136256 bytes 2023-12-04T14:23:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1246905344 bytes 2023-12-04T14:23:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1269830656 bytes 2023-12-04T14:23:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1283167232 bytes 2023-12-04T14:23:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1367281664 bytes 2023-12-04T14:23:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1487345664 bytes 2023-12-04T14:23:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1542505472 bytes 2023-12-04T14:23:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1591758848 bytes 2023-12-04T14:23:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1651577856 bytes 2023-12-04T14:23:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1740696576 bytes 2023-12-04T14:23:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1766224896 bytes 2023-12-04T14:23:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1780372480 bytes 2023-12-04T14:23:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1790995456 bytes 2023-12-04T14:23:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1866725376 bytes 2023-12-04T14:23:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1914903552 bytes 2023-12-04T14:23:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1936918528 bytes 2023-12-04T14:23:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1959523328 bytes 2023-12-04T14:23:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1968241664 bytes 2023-12-04T14:23:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1986837504 bytes 2023-12-04T14:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 2003198976 bytes 2023-12-04T14:23:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2017757184 bytes 2023-12-04T14:23:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2141085696 bytes 2023-12-04T14:23:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2162980864 bytes 2023-12-04T14:23:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2163582976 bytes 2023-12-04T14:23:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2210313216 bytes 2023-12-04T14:23:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2293629952 bytes 2023-12-04T14:23:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2296303616 bytes 2023-12-04T14:23:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2343302144 bytes 2023-12-04T14:23:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2371036160 bytes 2023-12-04T14:23:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2395470848 bytes 2023-12-04T14:23:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2396077056 bytes 2023-12-04T14:23:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2457177088 bytes 2023-12-04T14:23:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2474926080 bytes 2023-12-04T14:23:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2541390848 bytes 2023-12-04T14:23:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2610655232 bytes 2023-12-04T14:23:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2636522496 bytes 2023-12-04T14:23:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2689426432 bytes 2023-12-04T14:23:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2700480512 bytes 2023-12-04T14:23:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2725104640 bytes 2023-12-04T14:23:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2811692032 bytes 2023-12-04T14:23:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2900370432 bytes 2023-12-04T14:23:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2934079488 bytes 2023-12-04T14:23:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2991806464 bytes 2023-12-04T14:23:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:23:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:23:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:23:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:23:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:23:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_34 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_34 128M 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:23:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:23:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:23:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:23:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_34 2023-12-04T14:23:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:23:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:23:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:23:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-12-04T14:23:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:23:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:23:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-12-04T14:23:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:23:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-12-04T14:23:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_34': 2023-12-04T14:23:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:23:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:23:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:23:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 13efc220150d 2023-12-04T14:23:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.13efc220150d 2023-12-04T14:23:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:23:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:23:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:23:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:23:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:22 2023 2023-12-04T14:23:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:23:22 2023 2023-12-04T14:23:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:22 2023 2023-12-04T14:23:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:23:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:23:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 53abed9b-8d5d-4a15-a122-712b8dc9c3d5 2023-12-04T14:23:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:23:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:23:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_34 2023-12-04T14:23:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:23:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:23:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=12 2023-12-04T14:23:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:23:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:23:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 69123072 bytes 2023-12-04T14:23:23.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 86581248 bytes 2023-12-04T14:23:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 111319040 bytes 2023-12-04T14:23:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 188426240 bytes 2023-12-04T14:23:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 189063168 bytes 2023-12-04T14:23:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 189636608 bytes 2023-12-04T14:23:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 278649856 bytes 2023-12-04T14:23:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 295172096 bytes 2023-12-04T14:23:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 329158656 bytes 2023-12-04T14:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 373335040 bytes 2023-12-04T14:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 408430592 bytes 2023-12-04T14:23:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 470731776 bytes 2023-12-04T14:23:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 471356416 bytes 2023-12-04T14:23:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 529881088 bytes 2023-12-04T14:23:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 555920384 bytes 2023-12-04T14:23:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 577957888 bytes 2023-12-04T14:23:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 624581632 bytes 2023-12-04T14:23:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 637532160 bytes 2023-12-04T14:23:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 696550400 bytes 2023-12-04T14:23:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 729129984 bytes 2023-12-04T14:23:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 760228864 bytes 2023-12-04T14:23:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 847801344 bytes 2023-12-04T14:23:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 878166016 bytes 2023-12-04T14:23:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 978682880 bytes 2023-12-04T14:23:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1042101248 bytes 2023-12-04T14:23:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1105266688 bytes 2023-12-04T14:23:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1138299904 bytes 2023-12-04T14:23:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1181319168 bytes 2023-12-04T14:23:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1240644608 bytes 2023-12-04T14:23:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1280268288 bytes 2023-12-04T14:23:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1345929216 bytes 2023-12-04T14:23:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1391947776 bytes 2023-12-04T14:23:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1409378304 bytes 2023-12-04T14:23:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1439513600 bytes 2023-12-04T14:23:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1497490432 bytes 2023-12-04T14:23:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1534261248 bytes 2023-12-04T14:23:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1536768000 bytes 2023-12-04T14:23:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1593474048 bytes 2023-12-04T14:23:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1619056640 bytes 2023-12-04T14:23:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1684937728 bytes 2023-12-04T14:23:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1692040192 bytes 2023-12-04T14:23:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1719520256 bytes 2023-12-04T14:23:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1780614144 bytes 2023-12-04T14:23:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1808478208 bytes 2023-12-04T14:23:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1867323392 bytes 2023-12-04T14:23:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1921732608 bytes 2023-12-04T14:23:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2002002944 bytes 2023-12-04T14:23:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2059308032 bytes 2023-12-04T14:23:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2133518336 bytes 2023-12-04T14:23:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2134329344 bytes 2023-12-04T14:23:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2157045760 bytes 2023-12-04T14:23:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2218497024 bytes 2023-12-04T14:23:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2237048832 bytes 2023-12-04T14:23:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2247434240 bytes 2023-12-04T14:23:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2321529856 bytes 2023-12-04T14:23:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2402464768 bytes 2023-12-04T14:23:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2455499776 bytes 2023-12-04T14:23:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2532793344 bytes 2023-12-04T14:23:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2610832384 bytes 2023-12-04T14:23:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2674531328 bytes 2023-12-04T14:23:34.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2720162816 bytes 2023-12-04T14:23:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2787860480 bytes 2023-12-04T14:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_35 2023-12-04T14:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-12-04T14:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_35 128M 2023-12-04T14:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:23:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:23:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:23:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:23:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_35 2023-12-04T14:23:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:23:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:23:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:23:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-12-04T14:23:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:23:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:23:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-12-04T14:23:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:23:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-12-04T14:23:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_35': 2023-12-04T14:23:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:23:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:23:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:23:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 13fb6ed54c80 2023-12-04T14:23:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.13fb6ed54c80 2023-12-04T14:23:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:23:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:23:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:23:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:23:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:35 2023 2023-12-04T14:23:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:23:35 2023 2023-12-04T14:23:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:35 2023 2023-12-04T14:23:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:23:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:23:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 75dc1148-0c6b-4bf4-962c-9f4dc15daa97 2023-12-04T14:23:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:23:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:23:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_35 2023-12-04T14:23:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:23:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:23:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=7 2023-12-04T14:23:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:23:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:23:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 97968128 bytes 2023-12-04T14:23:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 126473216 bytes 2023-12-04T14:23:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 127090688 bytes 2023-12-04T14:23:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 149581824 bytes 2023-12-04T14:23:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 176305152 bytes 2023-12-04T14:23:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 198836224 bytes 2023-12-04T14:23:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 216034304 bytes 2023-12-04T14:23:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 257902592 bytes 2023-12-04T14:23:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 267657216 bytes 2023-12-04T14:23:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 324311040 bytes 2023-12-04T14:23:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 384857088 bytes 2023-12-04T14:23:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 447461376 bytes 2023-12-04T14:23:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 529428480 bytes 2023-12-04T14:23:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 546871296 bytes 2023-12-04T14:23:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 619310080 bytes 2023-12-04T14:23:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 638451712 bytes 2023-12-04T14:23:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 661442560 bytes 2023-12-04T14:23:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 701062144 bytes 2023-12-04T14:23:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 701674496 bytes 2023-12-04T14:23:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 727802880 bytes 2023-12-04T14:23:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 777990144 bytes 2023-12-04T14:23:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 805636096 bytes 2023-12-04T14:23:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 836886528 bytes 2023-12-04T14:23:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 861139968 bytes 2023-12-04T14:23:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 931578880 bytes 2023-12-04T14:23:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 986531840 bytes 2023-12-04T14:23:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1008053248 bytes 2023-12-04T14:23:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1021729792 bytes 2023-12-04T14:23:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1060468736 bytes 2023-12-04T14:23:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1116363776 bytes 2023-12-04T14:23:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1180704768 bytes 2023-12-04T14:23:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1190871040 bytes 2023-12-04T14:23:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1236217856 bytes 2023-12-04T14:23:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1283929088 bytes 2023-12-04T14:23:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1323760640 bytes 2023-12-04T14:23:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1353190400 bytes 2023-12-04T14:23:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1389079552 bytes 2023-12-04T14:23:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1438803968 bytes 2023-12-04T14:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1503597568 bytes 2023-12-04T14:23:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:23:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:23:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:23:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:23:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:23:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_36 2023-12-04T14:23:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-12-04T14:23:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:23:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:23:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:23:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_36 128M 2023-12-04T14:23:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:23:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:23:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:23:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:23:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_36 2023-12-04T14:23:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:23:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:23:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:23:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-12-04T14:23:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:23:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:23:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-12-04T14:23:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:23:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-12-04T14:23:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_36': 2023-12-04T14:23:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:23:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:23:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:23:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 14074396b83f 2023-12-04T14:23:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.14074396b83f 2023-12-04T14:23:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:23:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:23:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:23:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:23:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:43 2023 2023-12-04T14:23:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:23:43 2023 2023-12-04T14:23:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:43 2023 2023-12-04T14:23:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:23:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:23:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: c88f1ade-8455-43f9-be6e-45b69b6f15a1 2023-12-04T14:23:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:23:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:23:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_36 2023-12-04T14:23:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:23:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:23:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=12 2023-12-04T14:23:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:23:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:23:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 34674688 bytes 2023-12-04T14:23:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 59260928 bytes 2023-12-04T14:23:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 78881792 bytes 2023-12-04T14:23:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 122357760 bytes 2023-12-04T14:23:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 169867264 bytes 2023-12-04T14:23:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 223618048 bytes 2023-12-04T14:23:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 255584256 bytes 2023-12-04T14:23:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 348188672 bytes 2023-12-04T14:23:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 397828096 bytes 2023-12-04T14:23:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 445612032 bytes 2023-12-04T14:23:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 446253056 bytes 2023-12-04T14:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 535015424 bytes 2023-12-04T14:23:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 588808192 bytes 2023-12-04T14:23:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 646317056 bytes 2023-12-04T14:23:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 691879936 bytes 2023-12-04T14:23:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 714537984 bytes 2023-12-04T14:23:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 721719296 bytes 2023-12-04T14:23:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 820025344 bytes 2023-12-04T14:23:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 892016640 bytes 2023-12-04T14:23:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 902537216 bytes 2023-12-04T14:23:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 937851904 bytes 2023-12-04T14:23:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 943946752 bytes 2023-12-04T14:23:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 978101248 bytes 2023-12-04T14:23:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1016023040 bytes 2023-12-04T14:23:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1084282880 bytes 2023-12-04T14:23:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1181270016 bytes 2023-12-04T14:23:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1201483776 bytes 2023-12-04T14:23:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1235894272 bytes 2023-12-04T14:23:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1312443392 bytes 2023-12-04T14:23:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1379562496 bytes 2023-12-04T14:23:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1387832320 bytes 2023-12-04T14:23:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1399829504 bytes 2023-12-04T14:23:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1498544128 bytes 2023-12-04T14:23:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1545730048 bytes 2023-12-04T14:23:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1594832896 bytes 2023-12-04T14:23:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1634632704 bytes 2023-12-04T14:23:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1676996608 bytes 2023-12-04T14:23:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1715450880 bytes 2023-12-04T14:23:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1760262144 bytes 2023-12-04T14:23:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1772035072 bytes 2023-12-04T14:23:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1813257216 bytes 2023-12-04T14:23:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1836350464 bytes 2023-12-04T14:23:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1867922432 bytes 2023-12-04T14:23:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1889863680 bytes 2023-12-04T14:23:51.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1945561088 bytes 2023-12-04T14:23:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1981692928 bytes 2023-12-04T14:23:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2015470592 bytes 2023-12-04T14:23:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2040730624 bytes 2023-12-04T14:23:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2067428352 bytes 2023-12-04T14:23:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2135289856 bytes 2023-12-04T14:23:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2137835520 bytes 2023-12-04T14:23:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2174936064 bytes 2023-12-04T14:23:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2249233408 bytes 2023-12-04T14:23:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2309655552 bytes 2023-12-04T14:23:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2452817920 bytes 2023-12-04T14:23:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2582380544 bytes 2023-12-04T14:23:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2603601920 bytes 2023-12-04T14:23:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2612582400 bytes 2023-12-04T14:23:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2666109952 bytes 2023-12-04T14:23:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2716682240 bytes 2023-12-04T14:23:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2777689088 bytes 2023-12-04T14:23:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2857910272 bytes 2023-12-04T14:23:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:23:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:23:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:23:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:23:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:23:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_37 2023-12-04T14:23:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-12-04T14:23:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_37 128M 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:23:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:23:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:23:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:23:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:23:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_37 2023-12-04T14:23:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:23:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:23:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:23:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-12-04T14:23:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:23:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:23:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-12-04T14:23:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:23:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-12-04T14:23:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_37': 2023-12-04T14:23:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:23:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:23:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:23:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 141379811597 2023-12-04T14:23:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.141379811597 2023-12-04T14:23:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:23:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:23:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:23:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:23:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:55 2023 2023-12-04T14:23:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:23:55 2023 2023-12-04T14:23:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:55 2023 2023-12-04T14:23:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:23:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:23:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 573c9935-53e1-4562-b5e3-42d4b91a1ca2 2023-12-04T14:23:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:23:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:23:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_37 2023-12-04T14:23:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:23:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:23:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:23:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:23:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=7 2023-12-04T14:23:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:23:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:23:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 40710144 bytes 2023-12-04T14:23:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 65427456 bytes 2023-12-04T14:23:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 96770048 bytes 2023-12-04T14:23:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 97383424 bytes 2023-12-04T14:23:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 107765760 bytes 2023-12-04T14:23:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 163829760 bytes 2023-12-04T14:23:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 217759744 bytes 2023-12-04T14:23:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 301838336 bytes 2023-12-04T14:23:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 399294464 bytes 2023-12-04T14:23:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 433678336 bytes 2023-12-04T14:23:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 497094656 bytes 2023-12-04T14:23:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 534268928 bytes 2023-12-04T14:23:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 577959936 bytes 2023-12-04T14:23:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 619931648 bytes 2023-12-04T14:23:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 657854464 bytes 2023-12-04T14:23:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 662902784 bytes 2023-12-04T14:23:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 785132544 bytes 2023-12-04T14:23:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 818119680 bytes 2023-12-04T14:23:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 865551360 bytes 2023-12-04T14:23:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 917900288 bytes 2023-12-04T14:24:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 993301504 bytes 2023-12-04T14:24:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1047489536 bytes 2023-12-04T14:24:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1059329024 bytes 2023-12-04T14:24:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1111604224 bytes 2023-12-04T14:24:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1181719552 bytes 2023-12-04T14:24:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1219592192 bytes 2023-12-04T14:24:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1262237696 bytes 2023-12-04T14:24:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1287854080 bytes 2023-12-04T14:24:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1349964800 bytes 2023-12-04T14:24:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1354576896 bytes 2023-12-04T14:24:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1379821568 bytes 2023-12-04T14:24:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1473183744 bytes 2023-12-04T14:24:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1497269248 bytes 2023-12-04T14:24:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1534654464 bytes 2023-12-04T14:24:02.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1548862464 bytes 2023-12-04T14:24:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1570370560 bytes 2023-12-04T14:24:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1582396416 bytes 2023-12-04T14:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_38 2023-12-04T14:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-12-04T14:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_38 2023-12-04T14:24:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:24:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:24:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_38 128M 2023-12-04T14:24:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_38 2023-12-04T14:24:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:24:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:24:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:24:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_38 2023-12-04T14:24:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:24:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:24:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:24:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-12-04T14:24:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_38 2023-12-04T14:24:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:24:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-12-04T14:24:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:24:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-12-04T14:24:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_38': 2023-12-04T14:24:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:24:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:24:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:24:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 141e661dc075 2023-12-04T14:24:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.141e661dc075 2023-12-04T14:24:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:24:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:24:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:24:03 2023 2023-12-04T14:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:24:03 2023 2023-12-04T14:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:24:03 2023 2023-12-04T14:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: f19d0e0b-d80d-47ea-9448-f5c84f9d4cc6 2023-12-04T14:24:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:24:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:24:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_38 2023-12-04T14:24:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_38 2023-12-04T14:24:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:24:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=7 2023-12-04T14:24:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:24:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:24:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 28476416 bytes 2023-12-04T14:24:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 90724352 bytes 2023-12-04T14:24:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 153581568 bytes 2023-12-04T14:24:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 171284480 bytes 2023-12-04T14:24:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 274864128 bytes 2023-12-04T14:24:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 294242304 bytes 2023-12-04T14:24:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 417277952 bytes 2023-12-04T14:24:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 454123520 bytes 2023-12-04T14:24:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 486050816 bytes 2023-12-04T14:24:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 499068928 bytes 2023-12-04T14:24:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 610771968 bytes 2023-12-04T14:24:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 700777472 bytes 2023-12-04T14:24:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 790542336 bytes 2023-12-04T14:24:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 800285696 bytes 2023-12-04T14:24:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 856043520 bytes 2023-12-04T14:24:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 899158016 bytes 2023-12-04T14:24:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 993089536 bytes 2023-12-04T14:24:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 1076420608 bytes 2023-12-04T14:24:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 1092625408 bytes 2023-12-04T14:24:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 1116977152 bytes 2023-12-04T14:24:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 1154906112 bytes 2023-12-04T14:24:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1174200320 bytes 2023-12-04T14:24:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1218635776 bytes 2023-12-04T14:24:09.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1277275136 bytes 2023-12-04T14:24:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1315814400 bytes 2023-12-04T14:24:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1324461056 bytes 2023-12-04T14:24:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1368601600 bytes 2023-12-04T14:24:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1408891904 bytes 2023-12-04T14:24:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1484602368 bytes 2023-12-04T14:24:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1510813696 bytes 2023-12-04T14:24:10.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1538417664 bytes 2023-12-04T14:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1568105472 bytes 2023-12-04T14:24:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1615013888 bytes 2023-12-04T14:24:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1649409024 bytes 2023-12-04T14:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_39 2023-12-04T14:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-12-04T14:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_39 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_39 128M 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_39 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:24:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:24:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_39 2023-12-04T14:24:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:24:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:24:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:24:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-12-04T14:24:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_39 2023-12-04T14:24:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:24:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-12-04T14:24:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:24:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-12-04T14:24:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_39': 2023-12-04T14:24:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:24:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 142a6dd0e672 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.142a6dd0e672 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:24:11 2023 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:24:11 2023 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:24:11 2023 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 622dfb37-1a24-4c28-8ac0-9f49b29eeead 2023-12-04T14:24:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:24:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:24:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_39 2023-12-04T14:24:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_39 2023-12-04T14:24:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:24:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=14 2023-12-04T14:24:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:24:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:24:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 32366592 bytes 2023-12-04T14:24:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 70841344 bytes 2023-12-04T14:24:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 126095360 bytes 2023-12-04T14:24:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 156883968 bytes 2023-12-04T14:24:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 184911872 bytes 2023-12-04T14:24:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 191764480 bytes 2023-12-04T14:24:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 229187584 bytes 2023-12-04T14:24:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 290397184 bytes 2023-12-04T14:24:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 307573760 bytes 2023-12-04T14:24:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 322667520 bytes 2023-12-04T14:24:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 340518912 bytes 2023-12-04T14:24:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 389146624 bytes 2023-12-04T14:24:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 510091264 bytes 2023-12-04T14:24:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 561251328 bytes 2023-12-04T14:24:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 580415488 bytes 2023-12-04T14:24:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 620997632 bytes 2023-12-04T14:24:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 646482944 bytes 2023-12-04T14:24:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 657946624 bytes 2023-12-04T14:24:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 765173760 bytes 2023-12-04T14:24:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 826917888 bytes 2023-12-04T14:24:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 851716096 bytes 2023-12-04T14:24:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 855256064 bytes 2023-12-04T14:24:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 901170176 bytes 2023-12-04T14:24:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 918161408 bytes 2023-12-04T14:24:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1047186432 bytes 2023-12-04T14:24:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1057638400 bytes 2023-12-04T14:24:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1093447680 bytes 2023-12-04T14:24:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1153917952 bytes 2023-12-04T14:24:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1182808064 bytes 2023-12-04T14:24:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1234120704 bytes 2023-12-04T14:24:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1299822592 bytes 2023-12-04T14:24:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1353100288 bytes 2023-12-04T14:24:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1409798144 bytes 2023-12-04T14:24:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1439490048 bytes 2023-12-04T14:24:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1475415040 bytes 2023-12-04T14:24:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1477272576 bytes 2023-12-04T14:24:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1496394752 bytes 2023-12-04T14:24:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1527566336 bytes 2023-12-04T14:24:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1591600128 bytes 2023-12-04T14:24:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1628756992 bytes 2023-12-04T14:24:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1679052800 bytes 2023-12-04T14:24:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1712720896 bytes 2023-12-04T14:24:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1786206208 bytes 2023-12-04T14:24:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1823139840 bytes 2023-12-04T14:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1932115968 bytes 2023-12-04T14:24:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 2020711424 bytes 2023-12-04T14:24:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2050347008 bytes 2023-12-04T14:24:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2074280960 bytes 2023-12-04T14:24:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2081024000 bytes 2023-12-04T14:24:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2102724608 bytes 2023-12-04T14:24:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2148057088 bytes 2023-12-04T14:24:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2169851904 bytes 2023-12-04T14:24:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2195298304 bytes 2023-12-04T14:24:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2219352064 bytes 2023-12-04T14:24:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2230080512 bytes 2023-12-04T14:24:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2270795776 bytes 2023-12-04T14:24:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2322606080 bytes 2023-12-04T14:24:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2369238016 bytes 2023-12-04T14:24:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2406171648 bytes 2023-12-04T14:24:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2436442112 bytes 2023-12-04T14:24:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2468664320 bytes 2023-12-04T14:24:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2510839808 bytes 2023-12-04T14:24:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2537310208 bytes 2023-12-04T14:24:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2557326336 bytes 2023-12-04T14:24:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2584114176 bytes 2023-12-04T14:24:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2584736768 bytes 2023-12-04T14:24:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2596891648 bytes 2023-12-04T14:24:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2691014656 bytes 2023-12-04T14:24:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2702094336 bytes 2023-12-04T14:24:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 2806803456 bytes 2023-12-04T14:24:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 2865762304 bytes 2023-12-04T14:24:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 2910964736 bytes 2023-12-04T14:24:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7300 IOs, 2926358528 bytes 2023-12-04T14:24:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7400 IOs, 2952235008 bytes 2023-12-04T14:24:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7500 IOs, 3003758592 bytes 2023-12-04T14:24:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7600 IOs, 3066181632 bytes 2023-12-04T14:24:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7700 IOs, 3076517888 bytes 2023-12-04T14:24:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7800 IOs, 3131795456 bytes 2023-12-04T14:24:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:24:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:24:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:24:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:24:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_40 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_40 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_40 128M 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_40 2023-12-04T14:24:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:24:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:24:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:24:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_40 2023-12-04T14:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-12-04T14:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_40 2023-12-04T14:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:24:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-12-04T14:24:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:24:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-12-04T14:24:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_40': 2023-12-04T14:24:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:24:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:24:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:24:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 1436bc993d1e 2023-12-04T14:24:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.1436bc993d1e 2023-12-04T14:24:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:24:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:24:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:24:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:24:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:24:26 2023 2023-12-04T14:24:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:24:26 2023 2023-12-04T14:24:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:24:26 2023 2023-12-04T14:24:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:24:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:24:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: f1ae4816-50e9-48b7-a351-3bb0cf0cd5bb 2023-12-04T14:24:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:24:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:24:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_40 2023-12-04T14:24:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_40 2023-12-04T14:24:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:24:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=6 2023-12-04T14:24:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:24:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:24:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 12265472 bytes 2023-12-04T14:24:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 54491136 bytes 2023-12-04T14:24:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 78261248 bytes 2023-12-04T14:24:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 120591360 bytes 2023-12-04T14:24:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 149689344 bytes 2023-12-04T14:24:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 175483904 bytes 2023-12-04T14:24:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 211223552 bytes 2023-12-04T14:24:28.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 271182848 bytes 2023-12-04T14:24:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 288289792 bytes 2023-12-04T14:24:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 355709952 bytes 2023-12-04T14:24:28.895 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:24:28.895+0000 7f8e4ae66640 -1 reset not still connected to 0x55b1b68404e0 2023-12-04T14:24:28.895 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:24:28.895+0000 7f8e4ae66640 -1 reset not still connected to 0x55b1c3b11e10 2023-12-04T14:24:28.896 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:24:28.895+0000 7f8e4ae66640 -1 reset not still connected to 0x55b1ce4935f0 2023-12-04T14:24:28.896 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:24:28.895+0000 7f8e4ae66640 -1 reset not still connected to 0x55b1d22ddd40 2023-12-04T14:24:28.896 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:24:28.895+0000 7f8e4ae66640 -1 reset not still connected to 0x55b1d8bd1e10 2023-12-04T14:24:28.896 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:24:28.895+0000 7f8e4ae66640 -1 reset not still connected to 0x55b1e0323790 2023-12-04T14:24:28.896 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:24:28.895+0000 7f8e4ae66640 -1 reset not still connected to 0x55b1e6eca410 2023-12-04T14:24:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 387831808 bytes 2023-12-04T14:24:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 420306944 bytes 2023-12-04T14:24:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 439724032 bytes 2023-12-04T14:24:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 487095296 bytes 2023-12-04T14:24:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 505504768 bytes 2023-12-04T14:24:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 574594048 bytes 2023-12-04T14:24:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 598806528 bytes 2023-12-04T14:24:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 599397376 bytes 2023-12-04T14:24:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 637952000 bytes 2023-12-04T14:24:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 684281856 bytes 2023-12-04T14:24:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 708894720 bytes 2023-12-04T14:24:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 730304512 bytes 2023-12-04T14:24:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 828942336 bytes 2023-12-04T14:24:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 896740352 bytes 2023-12-04T14:24:31.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 959732736 bytes 2023-12-04T14:24:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 967951360 bytes 2023-12-04T14:24:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 974457856 bytes 2023-12-04T14:24:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1010580480 bytes 2023-12-04T14:24:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1118852096 bytes 2023-12-04T14:24:32.309 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-04T14:24:32.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-04T14:24:32.309 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-04T14:24:32.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi176.stderr:2023-12-04T14:24:32.308+0000 7f133a4a7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 101931) UID: 1000 2023-12-04T14:24:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1135355904 bytes 2023-12-04T14:24:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1135970304 bytes 2023-12-04T14:24:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1163746304 bytes 2023-12-04T14:24:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1167640576 bytes 2023-12-04T14:24:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1253343232 bytes 2023-12-04T14:24:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:24:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:24:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:24:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_41 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_41 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_41 128M 2023-12-04T14:24:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_41 2023-12-04T14:24:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:24:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:24:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:24:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_41 2023-12-04T14:24:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:24:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:24:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:24:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-12-04T14:24:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_41 2023-12-04T14:24:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:24:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-12-04T14:24:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:24:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-12-04T14:24:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_41': 2023-12-04T14:24:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 1442e42caa2e 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.1442e42caa2e 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:24:33 2023 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:24:33 2023 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:24:33 2023 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:24:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:24:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 4985d9fc-9c9b-4cb0-a17e-3e72f3e74408 2023-12-04T14:24:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:24:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:24:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_41 2023-12-04T14:24:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_41 2023-12-04T14:24:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:24:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=7 2023-12-04T14:24:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:24:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:24:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 15908864 bytes 2023-12-04T14:24:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 44069888 bytes 2023-12-04T14:24:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 58496000 bytes 2023-12-04T14:24:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 108591104 bytes 2023-12-04T14:24:34.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 166126592 bytes 2023-12-04T14:24:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 240311296 bytes 2023-12-04T14:24:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 270091264 bytes 2023-12-04T14:24:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 312762368 bytes 2023-12-04T14:24:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 396469248 bytes 2023-12-04T14:24:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 445583360 bytes 2023-12-04T14:24:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 513378304 bytes 2023-12-04T14:24:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 560152576 bytes 2023-12-04T14:24:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 600428544 bytes 2023-12-04T14:24:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 622763008 bytes 2023-12-04T14:24:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 626105344 bytes 2023-12-04T14:24:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 675986432 bytes 2023-12-04T14:24:37.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 727171072 bytes 2023-12-04T14:24:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 774964224 bytes 2023-12-04T14:24:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 838521856 bytes 2023-12-04T14:24:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 839103488 bytes 2023-12-04T14:24:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 884368384 bytes 2023-12-04T14:24:38.310 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-04T14:24:38.310 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:24:38.310 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-04T14:24:38.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-04T14:24:38.311 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-04T14:24:38.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-04T14:24:38.314 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-12-04T14:24:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 988630016 bytes 2023-12-04T14:24:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1035358208 bytes 2023-12-04T14:24:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1057006592 bytes 2023-12-04T14:24:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1084880896 bytes 2023-12-04T14:24:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1113553920 bytes 2023-12-04T14:24:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1114182656 bytes 2023-12-04T14:24:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1114818560 bytes 2023-12-04T14:24:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1130907648 bytes 2023-12-04T14:24:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1212210176 bytes 2023-12-04T14:24:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1216601088 bytes 2023-12-04T14:24:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1256135680 bytes 2023-12-04T14:24:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1313768448 bytes 2023-12-04T14:24:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1362948096 bytes 2023-12-04T14:24:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1416817664 bytes 2023-12-04T14:24:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1439716352 bytes 2023-12-04T14:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1532785664 bytes 2023-12-04T14:24:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:24:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:24:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:24:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:24:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:24:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_42 2023-12-04T14:24:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-12-04T14:24:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_42 2023-12-04T14:24:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:24:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:24:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_42 128M 2023-12-04T14:24:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_42 2023-12-04T14:24:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:24:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:24:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:24:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_42 2023-12-04T14:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-12-04T14:24:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_42 2023-12-04T14:24:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:24:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-12-04T14:24:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:24:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-12-04T14:24:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_42': 2023-12-04T14:24:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:24:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:24:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:24:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 1458bde53f75 2023-12-04T14:24:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.1458bde53f75 2023-12-04T14:24:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:24:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:24:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:24:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:24:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:24:41 2023 2023-12-04T14:24:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:24:41 2023 2023-12-04T14:24:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:24:41 2023 2023-12-04T14:24:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:24:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:24:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: fe0ec733-207f-4160-afad-a0d4633cee01 2023-12-04T14:24:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:24:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:24:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_42 2023-12-04T14:24:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_42 2023-12-04T14:24:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:24:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=12 2023-12-04T14:24:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:24:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:24:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 22823936 bytes 2023-12-04T14:24:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 110646272 bytes 2023-12-04T14:24:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 111250432 bytes 2023-12-04T14:24:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 126693376 bytes 2023-12-04T14:24:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 173626368 bytes 2023-12-04T14:24:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 207386624 bytes 2023-12-04T14:24:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 239802368 bytes 2023-12-04T14:24:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 309521408 bytes 2023-12-04T14:24:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 335804416 bytes 2023-12-04T14:24:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 400594944 bytes 2023-12-04T14:24:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 461040640 bytes 2023-12-04T14:24:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 494760960 bytes 2023-12-04T14:24:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 522798080 bytes 2023-12-04T14:24:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 556664832 bytes 2023-12-04T14:24:44.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 578851840 bytes 2023-12-04T14:24:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 644646912 bytes 2023-12-04T14:24:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 653925376 bytes 2023-12-04T14:24:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 654522368 bytes 2023-12-04T14:24:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 750653440 bytes 2023-12-04T14:24:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 820798464 bytes 2023-12-04T14:24:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 838769664 bytes 2023-12-04T14:24:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 891495424 bytes 2023-12-04T14:24:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 924279808 bytes 2023-12-04T14:24:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 945701888 bytes 2023-12-04T14:24:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 968112128 bytes 2023-12-04T14:24:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 983189504 bytes 2023-12-04T14:24:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1071555584 bytes 2023-12-04T14:24:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1096394752 bytes 2023-12-04T14:24:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1185163264 bytes 2023-12-04T14:24:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1194328064 bytes 2023-12-04T14:24:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1224255488 bytes 2023-12-04T14:24:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1308241920 bytes 2023-12-04T14:24:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1336359936 bytes 2023-12-04T14:24:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1368176640 bytes 2023-12-04T14:24:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1417880576 bytes 2023-12-04T14:24:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1485703168 bytes 2023-12-04T14:24:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1516816384 bytes 2023-12-04T14:24:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1598363648 bytes 2023-12-04T14:24:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1619935232 bytes 2023-12-04T14:24:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1672051712 bytes 2023-12-04T14:24:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1692507136 bytes 2023-12-04T14:24:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1725010944 bytes 2023-12-04T14:24:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1746695168 bytes 2023-12-04T14:24:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1794293760 bytes 2023-12-04T14:24:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1834221568 bytes 2023-12-04T14:24:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1852748800 bytes 2023-12-04T14:24:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1866499072 bytes 2023-12-04T14:24:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1959392256 bytes 2023-12-04T14:24:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2019441664 bytes 2023-12-04T14:24:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2042107904 bytes 2023-12-04T14:24:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2065408000 bytes 2023-12-04T14:24:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2076619776 bytes 2023-12-04T14:24:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2114881536 bytes 2023-12-04T14:24:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2133402624 bytes 2023-12-04T14:24:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2140000256 bytes 2023-12-04T14:24:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2179988480 bytes 2023-12-04T14:24:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2251193344 bytes 2023-12-04T14:24:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2274225152 bytes 2023-12-04T14:24:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2330765312 bytes 2023-12-04T14:24:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2331383808 bytes 2023-12-04T14:24:53.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2412568576 bytes 2023-12-04T14:24:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2452950016 bytes 2023-12-04T14:24:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2542427136 bytes 2023-12-04T14:24:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2556453888 bytes 2023-12-04T14:24:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2623909888 bytes 2023-12-04T14:24:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:24:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:24:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:24:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_43 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_43 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_43 128M 2023-12-04T14:24:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_43 2023-12-04T14:24:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:24:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:24:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:24:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:24:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_43 2023-12-04T14:24:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:24:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:24:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:24:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-12-04T14:24:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_43 2023-12-04T14:24:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:24:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-12-04T14:24:54.627 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7555066696213675) > weight (0.25) 2023-12-04T14:24:54.627 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.56937587342675) > weight (0.25) 2023-12-04T14:24:54.627 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-04T14:24:54.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-04T14:24:54.628 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-04T14:24:54.628 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-04T14:24:54.628 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:24:54.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr:2023-12-04T14:24:54.627+0000 7fb1967be640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 96665) UID: 1000 2023-12-04T14:24:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:24:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-12-04T14:24:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_43': 2023-12-04T14:24:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:24:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:24:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:24:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 14645f58beab 2023-12-04T14:24:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.14645f58beab 2023-12-04T14:24:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:24:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:24:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:24:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:24:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:24:54 2023 2023-12-04T14:24:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:24:54 2023 2023-12-04T14:24:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:24:54 2023 2023-12-04T14:24:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:24:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:24:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 5a31e773-c4a1-4a79-9d13-74cc228e5bc2 2023-12-04T14:24:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:24:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:24:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_43 2023-12-04T14:24:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:24:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:24:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_43 2023-12-04T14:24:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:24:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=12 2023-12-04T14:24:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:24:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:24:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 55730176 bytes 2023-12-04T14:24:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 95282176 bytes 2023-12-04T14:24:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 128107520 bytes 2023-12-04T14:24:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 178579456 bytes 2023-12-04T14:24:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 226448384 bytes 2023-12-04T14:24:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 255541248 bytes 2023-12-04T14:24:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 276706304 bytes 2023-12-04T14:24:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 291806208 bytes 2023-12-04T14:24:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 325735424 bytes 2023-12-04T14:24:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 369653760 bytes 2023-12-04T14:24:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 404280320 bytes 2023-12-04T14:24:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 446723072 bytes 2023-12-04T14:24:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 464120832 bytes 2023-12-04T14:24:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 464709632 bytes 2023-12-04T14:24:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 514058240 bytes 2023-12-04T14:24:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 514676736 bytes 2023-12-04T14:24:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 571541504 bytes 2023-12-04T14:24:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 588172288 bytes 2023-12-04T14:24:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 664545280 bytes 2023-12-04T14:24:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 712393728 bytes 2023-12-04T14:24:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 777571328 bytes 2023-12-04T14:24:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 797961216 bytes 2023-12-04T14:24:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 804779008 bytes 2023-12-04T14:24:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 835244032 bytes 2023-12-04T14:24:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 894162944 bytes 2023-12-04T14:24:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 938887168 bytes 2023-12-04T14:24:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1030174720 bytes 2023-12-04T14:25:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1065555968 bytes 2023-12-04T14:25:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1092374528 bytes 2023-12-04T14:25:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1124910080 bytes 2023-12-04T14:25:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1152386048 bytes 2023-12-04T14:25:00.730 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-04T14:25:00.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-04T14:25:00.730 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-04T14:25:00.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-04T14:25:00.733 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-12-04T14:25:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1217458176 bytes 2023-12-04T14:25:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1226720256 bytes 2023-12-04T14:25:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1259555840 bytes 2023-12-04T14:25:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1399263232 bytes 2023-12-04T14:25:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1482820608 bytes 2023-12-04T14:25:01.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1506859008 bytes 2023-12-04T14:25:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1533114368 bytes 2023-12-04T14:25:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1557928960 bytes 2023-12-04T14:25:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1564752896 bytes 2023-12-04T14:25:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1636865024 bytes 2023-12-04T14:25:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1716449280 bytes 2023-12-04T14:25:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1742516224 bytes 2023-12-04T14:25:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1784391680 bytes 2023-12-04T14:25:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1866361856 bytes 2023-12-04T14:25:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1954085888 bytes 2023-12-04T14:25:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2011100160 bytes 2023-12-04T14:25:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2078168064 bytes 2023-12-04T14:25:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2117876736 bytes 2023-12-04T14:25:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2165606400 bytes 2023-12-04T14:25:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2199533568 bytes 2023-12-04T14:25:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2253492224 bytes 2023-12-04T14:25:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2264154112 bytes 2023-12-04T14:25:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2296537088 bytes 2023-12-04T14:25:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2353968128 bytes 2023-12-04T14:25:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2362943488 bytes 2023-12-04T14:25:05.501 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:25:05.496+0000 7f4bfa770640 -1 reset not still connected to 0x56231a375040 2023-12-04T14:25:05.506 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:25:05.504+0000 7f4bfa770640 -1 reset not still connected to 0x56231a375860 2023-12-04T14:25:05.506 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:25:05.504+0000 7f4bfa770640 -1 reset not still connected to 0x56231a375c70 2023-12-04T14:25:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2393960448 bytes 2023-12-04T14:25:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2421169152 bytes 2023-12-04T14:25:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2452031488 bytes 2023-12-04T14:25:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2504408064 bytes 2023-12-04T14:25:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2545029120 bytes 2023-12-04T14:25:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2585555968 bytes 2023-12-04T14:25:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2640100352 bytes 2023-12-04T14:25:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2696524800 bytes 2023-12-04T14:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:25:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_44 2023-12-04T14:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-12-04T14:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_44 2023-12-04T14:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_44 128M 2023-12-04T14:25:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_44 2023-12-04T14:25:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:25:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:25:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:25:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_44 2023-12-04T14:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-12-04T14:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_44 2023-12-04T14:25:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:25:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-12-04T14:25:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:25:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-12-04T14:25:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_44': 2023-12-04T14:25:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:25:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:25:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:25:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 1475f547b45 2023-12-04T14:25:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.1475f547b45 2023-12-04T14:25:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:25:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:25:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:25:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:25:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:25:07 2023 2023-12-04T14:25:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:25:07 2023 2023-12-04T14:25:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:25:07 2023 2023-12-04T14:25:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:25:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:25:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 6705f07a-ea33-4f0a-b886-387067319889 2023-12-04T14:25:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:25:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:25:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_44 2023-12-04T14:25:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_44 2023-12-04T14:25:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:25:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=7 2023-12-04T14:25:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:25:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:25:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 41220096 bytes 2023-12-04T14:25:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 79853568 bytes 2023-12-04T14:25:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 112729088 bytes 2023-12-04T14:25:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 157589504 bytes 2023-12-04T14:25:09.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 209994752 bytes 2023-12-04T14:25:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 272581632 bytes 2023-12-04T14:25:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 292554752 bytes 2023-12-04T14:25:09.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 297607168 bytes 2023-12-04T14:25:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 340182016 bytes 2023-12-04T14:25:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 340808704 bytes 2023-12-04T14:25:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 368091136 bytes 2023-12-04T14:25:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 420499456 bytes 2023-12-04T14:25:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 504026112 bytes 2023-12-04T14:25:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 575571968 bytes 2023-12-04T14:25:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 613832704 bytes 2023-12-04T14:25:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 684046336 bytes 2023-12-04T14:25:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 718947328 bytes 2023-12-04T14:25:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 742176768 bytes 2023-12-04T14:25:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 774305792 bytes 2023-12-04T14:25:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 818322432 bytes 2023-12-04T14:25:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 825803776 bytes 2023-12-04T14:25:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 882956288 bytes 2023-12-04T14:25:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 940389376 bytes 2023-12-04T14:25:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 973744128 bytes 2023-12-04T14:25:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1019262976 bytes 2023-12-04T14:25:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1019886592 bytes 2023-12-04T14:25:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1077334016 bytes 2023-12-04T14:25:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1155676160 bytes 2023-12-04T14:25:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1206256640 bytes 2023-12-04T14:25:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1231436800 bytes 2023-12-04T14:25:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1272060928 bytes 2023-12-04T14:25:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1324011520 bytes 2023-12-04T14:25:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1385157632 bytes 2023-12-04T14:25:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1445483520 bytes 2023-12-04T14:25:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1469817856 bytes 2023-12-04T14:25:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1516367872 bytes 2023-12-04T14:25:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:25:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:25:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:25:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:25:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:25:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_45 2023-12-04T14:25:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-12-04T14:25:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_45 2023-12-04T14:25:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:25:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:25:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_45 128M 2023-12-04T14:25:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_45 2023-12-04T14:25:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:25:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:25:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:25:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_45 2023-12-04T14:25:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:25:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:25:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:25:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-12-04T14:25:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_45 2023-12-04T14:25:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:25:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-12-04T14:25:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:25:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-12-04T14:25:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_45': 2023-12-04T14:25:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:25:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:25:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:25:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 14817d155dfa 2023-12-04T14:25:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.14817d155dfa 2023-12-04T14:25:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:25:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:25:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:25:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:25:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:25:15 2023 2023-12-04T14:25:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:25:15 2023 2023-12-04T14:25:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:25:15 2023 2023-12-04T14:25:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:25:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:25:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: bd2f1aeb-d6bf-4777-b4de-b7dbba39dc7b 2023-12-04T14:25:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:25:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:25:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_45 2023-12-04T14:25:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_45 2023-12-04T14:25:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:25:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=6 2023-12-04T14:25:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:25:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:25:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 55387136 bytes 2023-12-04T14:25:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 126861312 bytes 2023-12-04T14:25:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 133554176 bytes 2023-12-04T14:25:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 166990848 bytes 2023-12-04T14:25:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 175793152 bytes 2023-12-04T14:25:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 190016512 bytes 2023-12-04T14:25:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 246518784 bytes 2023-12-04T14:25:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 289781760 bytes 2023-12-04T14:25:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 326812672 bytes 2023-12-04T14:25:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 333594624 bytes 2023-12-04T14:25:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 399984640 bytes 2023-12-04T14:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 464581632 bytes 2023-12-04T14:25:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 472581120 bytes 2023-12-04T14:25:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 525730816 bytes 2023-12-04T14:25:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 558947328 bytes 2023-12-04T14:25:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 621984768 bytes 2023-12-04T14:25:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 679857152 bytes 2023-12-04T14:25:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 791940096 bytes 2023-12-04T14:25:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 816109568 bytes 2023-12-04T14:25:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 869684224 bytes 2023-12-04T14:25:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 932983808 bytes 2023-12-04T14:25:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 964496384 bytes 2023-12-04T14:25:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 965577728 bytes 2023-12-04T14:25:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 966195200 bytes 2023-12-04T14:25:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1012950016 bytes 2023-12-04T14:25:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1063419904 bytes 2023-12-04T14:25:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1080239104 bytes 2023-12-04T14:25:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1135479808 bytes 2023-12-04T14:25:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1164020736 bytes 2023-12-04T14:25:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1221768192 bytes 2023-12-04T14:25:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1281588224 bytes 2023-12-04T14:25:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1306113024 bytes 2023-12-04T14:25:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1343462400 bytes 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_46 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_46 2023-12-04T14:25:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_46 128M 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_46 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_46 2023-12-04T14:25:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:25:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:25:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:25:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-12-04T14:25:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_46 2023-12-04T14:25:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:25:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-12-04T14:25:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:25:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-12-04T14:25:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_46': 2023-12-04T14:25:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:25:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:25:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:25:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 148de60844b5 2023-12-04T14:25:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.148de60844b5 2023-12-04T14:25:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:25:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:25:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:25:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:25:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:25:22 2023 2023-12-04T14:25:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:25:22 2023 2023-12-04T14:25:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:25:22 2023 2023-12-04T14:25:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:25:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:25:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: f4eea075-92cf-457e-8d7a-77a594f0d72d 2023-12-04T14:25:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:25:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:25:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_46 2023-12-04T14:25:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_46 2023-12-04T14:25:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:25:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=14 2023-12-04T14:25:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:25:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:25:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 64414720 bytes 2023-12-04T14:25:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 84454400 bytes 2023-12-04T14:25:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 91913216 bytes 2023-12-04T14:25:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 104956928 bytes 2023-12-04T14:25:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 148015104 bytes 2023-12-04T14:25:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 162682880 bytes 2023-12-04T14:25:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 200382464 bytes 2023-12-04T14:25:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 245187584 bytes 2023-12-04T14:25:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 295367680 bytes 2023-12-04T14:25:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 318291968 bytes 2023-12-04T14:25:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 362601472 bytes 2023-12-04T14:25:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 428013568 bytes 2023-12-04T14:25:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 503418880 bytes 2023-12-04T14:25:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 577316864 bytes 2023-12-04T14:25:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 607955968 bytes 2023-12-04T14:25:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 613647360 bytes 2023-12-04T14:25:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 632120320 bytes 2023-12-04T14:25:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 700189696 bytes 2023-12-04T14:25:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 792482816 bytes 2023-12-04T14:25:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 817105920 bytes 2023-12-04T14:25:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 833182720 bytes 2023-12-04T14:25:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 871048192 bytes 2023-12-04T14:25:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 950463488 bytes 2023-12-04T14:25:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1002894336 bytes 2023-12-04T14:25:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1039907840 bytes 2023-12-04T14:25:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1129232384 bytes 2023-12-04T14:25:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1171963904 bytes 2023-12-04T14:25:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1207751680 bytes 2023-12-04T14:25:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1256861696 bytes 2023-12-04T14:25:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1326694400 bytes 2023-12-04T14:25:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1348929536 bytes 2023-12-04T14:25:29.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1379214336 bytes 2023-12-04T14:25:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1379849216 bytes 2023-12-04T14:25:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1438345216 bytes 2023-12-04T14:25:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1479455744 bytes 2023-12-04T14:25:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1480058880 bytes 2023-12-04T14:25:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1540468736 bytes 2023-12-04T14:25:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1589178368 bytes 2023-12-04T14:25:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1612049408 bytes 2023-12-04T14:25:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1700267008 bytes 2023-12-04T14:25:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1809498112 bytes 2023-12-04T14:25:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1820796928 bytes 2023-12-04T14:25:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1861100544 bytes 2023-12-04T14:25:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1934494720 bytes 2023-12-04T14:25:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1983666176 bytes 2023-12-04T14:25:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 2028520448 bytes 2023-12-04T14:25:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2060889088 bytes 2023-12-04T14:25:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2083936256 bytes 2023-12-04T14:25:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2096807936 bytes 2023-12-04T14:25:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2163397632 bytes 2023-12-04T14:25:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2218121216 bytes 2023-12-04T14:25:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2260251648 bytes 2023-12-04T14:25:33.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2325070848 bytes 2023-12-04T14:25:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2361787392 bytes 2023-12-04T14:25:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2389432320 bytes 2023-12-04T14:25:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2407853056 bytes 2023-12-04T14:25:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2454553600 bytes 2023-12-04T14:25:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2479761408 bytes 2023-12-04T14:25:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2549774336 bytes 2023-12-04T14:25:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2550367232 bytes 2023-12-04T14:25:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2571829248 bytes 2023-12-04T14:25:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2605972480 bytes 2023-12-04T14:25:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2621198336 bytes 2023-12-04T14:25:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2653881344 bytes 2023-12-04T14:25:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2666850304 bytes 2023-12-04T14:25:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2706022400 bytes 2023-12-04T14:25:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2766092288 bytes 2023-12-04T14:25:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2815998976 bytes 2023-12-04T14:25:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2939286528 bytes 2023-12-04T14:25:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 2988537856 bytes 2023-12-04T14:25:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 3052593152 bytes 2023-12-04T14:25:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 3130313728 bytes 2023-12-04T14:25:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7300 IOs, 3158282240 bytes 2023-12-04T14:25:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:25:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:25:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:25:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:25:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:25:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_47 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_47 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_47 128M 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_47 2023-12-04T14:25:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:25:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:25:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:25:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_47 2023-12-04T14:25:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:25:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:25:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:25:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-12-04T14:25:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_47 2023-12-04T14:25:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:25:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-12-04T14:25:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:25:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-12-04T14:25:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_47': 2023-12-04T14:25:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:25:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:25:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:25:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 14995f04fb5d 2023-12-04T14:25:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.14995f04fb5d 2023-12-04T14:25:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:25:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:25:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:25:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:25:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:25:37 2023 2023-12-04T14:25:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:25:37 2023 2023-12-04T14:25:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:25:37 2023 2023-12-04T14:25:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:25:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:25:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 1c9b05a7-06ae-420c-ab01-732da45bf605 2023-12-04T14:25:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:25:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:25:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_47 2023-12-04T14:25:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_47 2023-12-04T14:25:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:25:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=13 2023-12-04T14:25:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:25:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:25:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 28557312 bytes 2023-12-04T14:25:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 42202112 bytes 2023-12-04T14:25:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 88382464 bytes 2023-12-04T14:25:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 127350784 bytes 2023-12-04T14:25:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 176138240 bytes 2023-12-04T14:25:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 228235264 bytes 2023-12-04T14:25:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 298341376 bytes 2023-12-04T14:25:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 353540096 bytes 2023-12-04T14:25:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 406665216 bytes 2023-12-04T14:25:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 516844544 bytes 2023-12-04T14:25:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 532347904 bytes 2023-12-04T14:25:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 608626688 bytes 2023-12-04T14:25:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 675505152 bytes 2023-12-04T14:25:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 686078976 bytes 2023-12-04T14:25:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 715615232 bytes 2023-12-04T14:25:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 795533312 bytes 2023-12-04T14:25:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 835117056 bytes 2023-12-04T14:25:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 866383872 bytes 2023-12-04T14:25:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 914809856 bytes 2023-12-04T14:25:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 996588544 bytes 2023-12-04T14:25:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 1019197440 bytes 2023-12-04T14:25:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1064939520 bytes 2023-12-04T14:25:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1083000832 bytes 2023-12-04T14:25:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1108023296 bytes 2023-12-04T14:25:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1152750592 bytes 2023-12-04T14:25:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1188214784 bytes 2023-12-04T14:25:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1236917248 bytes 2023-12-04T14:25:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1296449536 bytes 2023-12-04T14:25:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1316971520 bytes 2023-12-04T14:25:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1324227584 bytes 2023-12-04T14:25:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1327929344 bytes 2023-12-04T14:25:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1368807424 bytes 2023-12-04T14:25:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1441397760 bytes 2023-12-04T14:25:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1491385344 bytes 2023-12-04T14:25:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1491987456 bytes 2023-12-04T14:25:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1541894144 bytes 2023-12-04T14:25:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1610855424 bytes 2023-12-04T14:25:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1632970752 bytes 2023-12-04T14:25:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1650513920 bytes 2023-12-04T14:25:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1698159616 bytes 2023-12-04T14:25:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1736003584 bytes 2023-12-04T14:25:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1755433984 bytes 2023-12-04T14:25:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1802577920 bytes 2023-12-04T14:25:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1824017408 bytes 2023-12-04T14:25:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1857409024 bytes 2023-12-04T14:25:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1878723584 bytes 2023-12-04T14:25:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1914383360 bytes 2023-12-04T14:25:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1928949760 bytes 2023-12-04T14:25:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 1985499136 bytes 2023-12-04T14:25:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2032520192 bytes 2023-12-04T14:25:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2071951360 bytes 2023-12-04T14:25:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2107675648 bytes 2023-12-04T14:25:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2150781952 bytes 2023-12-04T14:25:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2180670464 bytes 2023-12-04T14:25:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2239539200 bytes 2023-12-04T14:25:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2256191488 bytes 2023-12-04T14:25:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2306947072 bytes 2023-12-04T14:25:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2351513600 bytes 2023-12-04T14:25:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2393699328 bytes 2023-12-04T14:25:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2437452800 bytes 2023-12-04T14:25:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2475941888 bytes 2023-12-04T14:25:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2545779712 bytes 2023-12-04T14:25:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2552461312 bytes 2023-12-04T14:25:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2575716352 bytes 2023-12-04T14:25:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2603154432 bytes 2023-12-04T14:25:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 2617772032 bytes 2023-12-04T14:25:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 2698027008 bytes 2023-12-04T14:25:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 2738046976 bytes 2023-12-04T14:25:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 2777205760 bytes 2023-12-04T14:25:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 2794584064 bytes 2023-12-04T14:25:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 2815421440 bytes 2023-12-04T14:25:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 2852912128 bytes 2023-12-04T14:25:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:25:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:25:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:25:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:25:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:25:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_48 2023-12-04T14:25:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-12-04T14:25:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_48 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_48 128M 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_48 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:25:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:25:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:25:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:25:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_48 2023-12-04T14:25:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:25:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:25:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:25:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-12-04T14:25:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_48 2023-12-04T14:25:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:25:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-12-04T14:25:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:25:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-12-04T14:25:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_48': 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 14a5cf319ee8 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.14a5cf319ee8 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:25:51 2023 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:25:51 2023 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:25:51 2023 2023-12-04T14:25:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:25:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:25:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 75662a66-3ee6-422c-bc5b-2ed60a3a9402 2023-12-04T14:25:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:25:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:25:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_48 2023-12-04T14:25:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:25:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:25:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_48 2023-12-04T14:25:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:25:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=10 2023-12-04T14:25:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:25:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:25:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 48864256 bytes 2023-12-04T14:25:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 103786496 bytes 2023-12-04T14:25:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 136112128 bytes 2023-12-04T14:25:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 178101248 bytes 2023-12-04T14:25:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 250923008 bytes 2023-12-04T14:25:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 349422592 bytes 2023-12-04T14:25:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 382187520 bytes 2023-12-04T14:25:53.812 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:25:53.810+0000 7f8e47a49640 -1 bdev(0x55b19f8b1180 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-12-04T14:25:53.814 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:25:53.812+0000 7f8e4f263640 -1 bdev(0x55b19f8b0e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-04T14:25:53.814 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:25:53.812+0000 7f8e47a49640 -1 bdev(0x55b19f8b1180 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-12-04T14:25:53.815 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7672-g7c730b29/rpm/el9/BUILD/ceph-18.0.0-7672-g7c730b29/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f8e4f263640 time 2023-12-04T14:25:53.813328+0000 2023-12-04T14:25:53.815 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7672-g7c730b29/rpm/el9/BUILD/ceph-18.0.0-7672-g7c730b29/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-04T14:25:53.815 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:25:53.815 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55b19c24f33b] 2023-12-04T14:25:53.815 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x55b19cba5f06] 2023-12-04T14:25:53.815 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 3: ceph-osd(+0xdc7071) [0x55b19cba6071] 2023-12-04T14:25:53.815 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f8e5c89f802] 2023-12-04T14:25:53.815 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f8e5c83f450] 2023-12-04T14:25:53.815 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:*** Caught signal (Aborted) ** 2023-12-04T14:25:53.815 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: in thread 7f8e4f263640 thread_name:bstore_aio 2023-12-04T14:25:53.815 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:25:53.813+0000 7f8e4f263640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7672-g7c730b29/rpm/el9/BUILD/ceph-18.0.0-7672-g7c730b29/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f8e4f263640 time 2023-12-04T14:25:53.813328+0000 2023-12-04T14:25:53.816 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7672-g7c730b29/rpm/el9/BUILD/ceph-18.0.0-7672-g7c730b29/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-04T14:25:53.816 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2023-12-04T14:25:53.816 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:25:53.816 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55b19c24f33b] 2023-12-04T14:25:53.816 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x55b19cba5f06] 2023-12-04T14:25:53.816 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 3: ceph-osd(+0xdc7071) [0x55b19cba6071] 2023-12-04T14:25:53.816 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f8e5c89f802] 2023-12-04T14:25:53.816 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f8e5c83f450] 2023-12-04T14:25:53.816 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f8e5c854db0] 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f8e5c8a154c] 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 3: raise() 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 4: abort() 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55b19c24f3fc] 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x55b19cba5f06] 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 7: ceph-osd(+0xdc7071) [0x55b19cba6071] 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f8e5c89f802] 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f8e5c83f450] 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:2023-12-04T14:25:53.814+0000 7f8e4f263640 -1 *** Caught signal (Aborted) ** 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: in thread 7f8e4f263640 thread_name:bstore_aio 2023-12-04T14:25:53.817 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2023-12-04T14:25:53.818 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:25:53.818 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f8e5c854db0] 2023-12-04T14:25:53.818 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f8e5c8a154c] 2023-12-04T14:25:53.818 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 3: raise() 2023-12-04T14:25:53.818 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 4: abort() 2023-12-04T14:25:53.818 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55b19c24f3fc] 2023-12-04T14:25:53.818 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x55b19cba5f06] 2023-12-04T14:25:53.818 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 7: ceph-osd(+0xdc7071) [0x55b19cba6071] 2023-12-04T14:25:53.818 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f8e5c89f802] 2023-12-04T14:25:53.818 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f8e5c83f450] 2023-12-04T14:25:53.818 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T14:25:53.818 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2023-12-04T14:25:53.832 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: -348> 2023-12-04T14:25:53.810+0000 7f8e47a49640 -1 bdev(0x55b19f8b1180 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-12-04T14:25:53.832 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: -165> 2023-12-04T14:25:53.812+0000 7f8e4f263640 -1 bdev(0x55b19f8b0e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-04T14:25:53.832 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: -109> 2023-12-04T14:25:53.812+0000 7f8e47a49640 -1 bdev(0x55b19f8b1180 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: -31> 2023-12-04T14:25:53.813+0000 7f8e4f263640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7672-g7c730b29/rpm/el9/BUILD/ceph-18.0.0-7672-g7c730b29/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f8e4f263640 time 2023-12-04T14:25:53.813328+0000 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7672-g7c730b29/rpm/el9/BUILD/ceph-18.0.0-7672-g7c730b29/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55b19c24f33b] 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x55b19cba5f06] 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 3: ceph-osd(+0xdc7071) [0x55b19cba6071] 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f8e5c89f802] 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f8e5c83f450] 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 0> 2023-12-04T14:25:53.814+0000 7f8e4f263640 -1 *** Caught signal (Aborted) ** 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: in thread 7f8e4f263640 thread_name:bstore_aio 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2023-12-04T14:25:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:25:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f8e5c854db0] 2023-12-04T14:25:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f8e5c8a154c] 2023-12-04T14:25:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 3: raise() 2023-12-04T14:25:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 4: abort() 2023-12-04T14:25:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55b19c24f3fc] 2023-12-04T14:25:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x55b19cba5f06] 2023-12-04T14:25:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 7: ceph-osd(+0xdc7071) [0x55b19cba6071] 2023-12-04T14:25:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f8e5c89f802] 2023-12-04T14:25:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f8e5c83f450] 2023-12-04T14:25:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T14:25:53.834 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2023-12-04T14:25:53.842 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: -614> 2023-12-04T14:25:53.810+0000 7f8e47a49640 -1 bdev(0x55b19f8b1180 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-12-04T14:25:53.842 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: -431> 2023-12-04T14:25:53.812+0000 7f8e4f263640 -1 bdev(0x55b19f8b0e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-04T14:25:53.842 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: -375> 2023-12-04T14:25:53.812+0000 7f8e47a49640 -1 bdev(0x55b19f8b1180 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-12-04T14:25:53.842 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: -297> 2023-12-04T14:25:53.813+0000 7f8e4f263640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7672-g7c730b29/rpm/el9/BUILD/ceph-18.0.0-7672-g7c730b29/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f8e4f263640 time 2023-12-04T14:25:53.813328+0000 2023-12-04T14:25:53.842 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7672-g7c730b29/rpm/el9/BUILD/ceph-18.0.0-7672-g7c730b29/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-04T14:25:53.842 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55b19c24f33b] 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x55b19cba5f06] 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 3: ceph-osd(+0xdc7071) [0x55b19cba6071] 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f8e5c89f802] 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f8e5c83f450] 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: -266> 2023-12-04T14:25:53.814+0000 7f8e4f263640 -1 *** Caught signal (Aborted) ** 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: in thread 7f8e4f263640 thread_name:bstore_aio 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f8e5c854db0] 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f8e5c8a154c] 2023-12-04T14:25:53.843 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 3: raise() 2023-12-04T14:25:53.844 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 4: abort() 2023-12-04T14:25:53.844 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55b19c24f3fc] 2023-12-04T14:25:53.844 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x55b19cba5f06] 2023-12-04T14:25:53.844 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 7: ceph-osd(+0xdc7071) [0x55b19cba6071] 2023-12-04T14:25:53.844 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f8e5c89f802] 2023-12-04T14:25:53.844 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f8e5c83f450] 2023-12-04T14:25:53.844 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T14:25:53.844 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr: 2023-12-04T14:25:53.929 INFO:tasks.ceph.cluster2.osd.0.smithi176.stderr:daemon-helper: command failed with exit status 5 2023-12-04T14:25:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 421627904 bytes 2023-12-04T14:25:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 486045696 bytes 2023-12-04T14:25:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 589082624 bytes 2023-12-04T14:25:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 660885504 bytes 2023-12-04T14:25:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 700510208 bytes 2023-12-04T14:25:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 731210752 bytes 2023-12-04T14:25:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 771104768 bytes 2023-12-04T14:25:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 809477120 bytes 2023-12-04T14:25:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 810107904 bytes 2023-12-04T14:25:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 858587136 bytes 2023-12-04T14:25:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 869641216 bytes 2023-12-04T14:25:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 897746944 bytes 2023-12-04T14:25:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 921075712 bytes 2023-12-04T14:25:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 977970176 bytes 2023-12-04T14:25:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1042453504 bytes 2023-12-04T14:25:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1086356480 bytes 2023-12-04T14:25:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1171496960 bytes 2023-12-04T14:25:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1196460032 bytes 2023-12-04T14:25:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1218246656 bytes 2023-12-04T14:25:57.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1237500928 bytes 2023-12-04T14:25:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1259658240 bytes 2023-12-04T14:25:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1327974400 bytes 2023-12-04T14:25:58.148 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-12-04T14:25:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1356550144 bytes 2023-12-04T14:25:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1423553536 bytes 2023-12-04T14:25:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1500882944 bytes 2023-12-04T14:25:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1551938560 bytes 2023-12-04T14:25:58.879 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:25:58.877+0000 7f78b9196640 -1 reset not still connected to 0x556ae8c49110 2023-12-04T14:25:58.879 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:25:58.877+0000 7f78b9196640 -1 reset not still connected to 0x556ae8c49c70 2023-12-04T14:25:58.879 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:25:58.878+0000 7f78b9196640 -1 reset not still connected to 0x556aeaf46410 2023-12-04T14:25:58.879 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:25:58.878+0000 7f78b9196640 -1 reset not still connected to 0x556aeaf465b0 2023-12-04T14:25:58.880 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:25:58.878+0000 7f78b9196640 -1 reset not still connected to 0x556aebd0aea0 2023-12-04T14:25:58.880 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:25:58.878+0000 7f78b9196640 -1 reset not still connected to 0x556aec844270 2023-12-04T14:25:58.880 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:25:58.878+0000 7f78b9196640 -1 reset not still connected to 0x556aec943520 2023-12-04T14:25:58.880 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:25:58.878+0000 7f78b9196640 -1 reset not still connected to 0x556aec9436c0 2023-12-04T14:25:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1605729280 bytes 2023-12-04T14:25:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1654255616 bytes 2023-12-04T14:25:59.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1654851584 bytes 2023-12-04T14:25:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1682219008 bytes 2023-12-04T14:25:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1709125632 bytes 2023-12-04T14:25:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1729342464 bytes 2023-12-04T14:25:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1732659200 bytes 2023-12-04T14:25:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1771900928 bytes 2023-12-04T14:25:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1819972608 bytes 2023-12-04T14:25:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1852116992 bytes 2023-12-04T14:26:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 1860581376 bytes 2023-12-04T14:26:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 1878068224 bytes 2023-12-04T14:26:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 1884683264 bytes 2023-12-04T14:26:00.354 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:26:00.352+0000 7f4bfa770640 -1 reset not still connected to 0x56233d93eb60 2023-12-04T14:26:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 1948858368 bytes 2023-12-04T14:26:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 1995042816 bytes 2023-12-04T14:26:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2048959488 bytes 2023-12-04T14:26:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2064408576 bytes 2023-12-04T14:26:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2121101312 bytes 2023-12-04T14:26:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2214626304 bytes 2023-12-04T14:26:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2232051712 bytes 2023-12-04T14:26:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2282852352 bytes 2023-12-04T14:26:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2288845824 bytes 2023-12-04T14:26:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2318570496 bytes 2023-12-04T14:26:01.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2355034112 bytes 2023-12-04T14:26:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2424655872 bytes 2023-12-04T14:26:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2484561920 bytes 2023-12-04T14:26:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:26:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:26:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:26:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:26:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_49 2023-12-04T14:26:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-12-04T14:26:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_49 2023-12-04T14:26:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:26:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:26:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_49 128M 2023-12-04T14:26:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_49 2023-12-04T14:26:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:26:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:26:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:26:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_49 2023-12-04T14:26:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:26:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:26:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-12-04T14:26:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_49 2023-12-04T14:26:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:26:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-12-04T14:26:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:26:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-12-04T14:26:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_49': 2023-12-04T14:26:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:26:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:26:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:26:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 14b0c661fc24 2023-12-04T14:26:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.14b0c661fc24 2023-12-04T14:26:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:26:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:26:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:26:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:26:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:26:02 2023 2023-12-04T14:26:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:26:02 2023 2023-12-04T14:26:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:26:02 2023 2023-12-04T14:26:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:26:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:26:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: c368a0ff-abe8-4e7e-bf74-d33daf9fed3d 2023-12-04T14:26:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:26:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:26:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_49 2023-12-04T14:26:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_49 2023-12-04T14:26:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=5 2023-12-04T14:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:26:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 41661440 bytes 2023-12-04T14:26:03.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-12-04T14:26:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 59485184 bytes 2023-12-04T14:26:03.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 95869952 bytes 2023-12-04T14:26:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 111342592 bytes 2023-12-04T14:26:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 157776896 bytes 2023-12-04T14:26:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 161358848 bytes 2023-12-04T14:26:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 174155776 bytes 2023-12-04T14:26:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 261978112 bytes 2023-12-04T14:26:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 334053376 bytes 2023-12-04T14:26:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 385599488 bytes 2023-12-04T14:26:04.733 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-04T14:26:04.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-04T14:26:04.734 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-04T14:26:04.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi176.stderr:2023-12-04T14:26:04.733+0000 7f766892a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 100728) UID: 1000 2023-12-04T14:26:04.742 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 480391168 bytes 2023-12-04T14:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 497592320 bytes 2023-12-04T14:26:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 555766784 bytes 2023-12-04T14:26:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 630290432 bytes 2023-12-04T14:26:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 640362496 bytes 2023-12-04T14:26:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 685325312 bytes 2023-12-04T14:26:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 718664704 bytes 2023-12-04T14:26:05.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 750120960 bytes 2023-12-04T14:26:05.735 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-04T14:26:05.735 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:26:05.736 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-04T14:26:05.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-04T14:26:05.736 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-04T14:26:05.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-04T14:26:05.739 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2023-12-04T14:26:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 863857664 bytes 2023-12-04T14:26:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 887685120 bytes 2023-12-04T14:26:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 945966080 bytes 2023-12-04T14:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 1005237248 bytes 2023-12-04T14:26:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1066387456 bytes 2023-12-04T14:26:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1126776832 bytes 2023-12-04T14:26:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1174530048 bytes 2023-12-04T14:26:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1197254656 bytes 2023-12-04T14:26:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1241368576 bytes 2023-12-04T14:26:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1284148224 bytes 2023-12-04T14:26:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1390973952 bytes 2023-12-04T14:26:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1410620416 bytes 2023-12-04T14:26:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1422834688 bytes 2023-12-04T14:26:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1444251648 bytes 2023-12-04T14:26:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1505459200 bytes 2023-12-04T14:26:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1622616064 bytes 2023-12-04T14:26:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1650451456 bytes 2023-12-04T14:26:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1732662272 bytes 2023-12-04T14:26:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1754497024 bytes 2023-12-04T14:26:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:26:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:26:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:26:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:26:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_50 2023-12-04T14:26:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-12-04T14:26:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_50 2023-12-04T14:26:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:26:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n snapshot ']' 2023-12-04T14:26:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_image cluster2 mirror image_50 128M 2023-12-04T14:26:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_50 2023-12-04T14:26:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local size=128 2023-12-04T14:26:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n 128M ']' 2023-12-04T14:26:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ size=128M 2023-12-04T14:26:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ shift 2023-12-04T14:26:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_50 2023-12-04T14:26:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ is_pool_mirror_mode_image mirror 2023-12-04T14:26:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' image = image ']' 2023-12-04T14:26:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-12-04T14:26:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_50 2023-12-04T14:26:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local mode=snapshot 2023-12-04T14:26:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-12-04T14:26:08.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-12-04T14:26:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirroring enabled 2023-12-04T14:26:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-12-04T14:26:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_50': 2023-12-04T14:26:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:26:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:26:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:26:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 14bede40fe7 2023-12-04T14:26:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.14bede40fe7 2023-12-04T14:26:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering 2023-12-04T14:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:26:08 2023 2023-12-04T14:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:26:08 2023 2023-12-04T14:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:26:08 2023 2023-12-04T14:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: b6c8de6f-5527-43d7-be14-65c94f60c736 2023-12-04T14:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: true 2023-12-04T14:26:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -n '' ']' 2023-12-04T14:26:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stress_write_image cluster2 mirror image_50 2023-12-04T14:26:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_50 2023-12-04T14:26:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-04T14:26:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local duration=14 2023-12-04T14:26:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:26:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-12-04T14:26:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:100 IOs, 24024064 bytes 2023-12-04T14:26:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:200 IOs, 74608640 bytes 2023-12-04T14:26:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:300 IOs, 127225856 bytes 2023-12-04T14:26:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:400 IOs, 168373248 bytes 2023-12-04T14:26:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:500 IOs, 243225600 bytes 2023-12-04T14:26:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:600 IOs, 243818496 bytes 2023-12-04T14:26:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:700 IOs, 300125184 bytes 2023-12-04T14:26:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:800 IOs, 365706240 bytes 2023-12-04T14:26:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:900 IOs, 406281216 bytes 2023-12-04T14:26:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1000 IOs, 530299904 bytes 2023-12-04T14:26:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1100 IOs, 543073280 bytes 2023-12-04T14:26:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1200 IOs, 569079808 bytes 2023-12-04T14:26:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1300 IOs, 612307968 bytes 2023-12-04T14:26:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1400 IOs, 645342208 bytes 2023-12-04T14:26:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1500 IOs, 705086464 bytes 2023-12-04T14:26:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1600 IOs, 705732608 bytes 2023-12-04T14:26:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1700 IOs, 742141952 bytes 2023-12-04T14:26:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1800 IOs, 785087488 bytes 2023-12-04T14:26:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:1900 IOs, 842076160 bytes 2023-12-04T14:26:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2000 IOs, 918992896 bytes 2023-12-04T14:26:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2100 IOs, 930058240 bytes 2023-12-04T14:26:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2200 IOs, 998892544 bytes 2023-12-04T14:26:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2300 IOs, 1089776640 bytes 2023-12-04T14:26:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2400 IOs, 1174133760 bytes 2023-12-04T14:26:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2500 IOs, 1244394496 bytes 2023-12-04T14:26:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2600 IOs, 1300408320 bytes 2023-12-04T14:26:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2700 IOs, 1345598464 bytes 2023-12-04T14:26:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2800 IOs, 1372384256 bytes 2023-12-04T14:26:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:2900 IOs, 1426825216 bytes 2023-12-04T14:26:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3000 IOs, 1467172864 bytes 2023-12-04T14:26:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3100 IOs, 1517838336 bytes 2023-12-04T14:26:13.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3200 IOs, 1554736128 bytes 2023-12-04T14:26:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3300 IOs, 1579672576 bytes 2023-12-04T14:26:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3400 IOs, 1610119168 bytes 2023-12-04T14:26:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3500 IOs, 1671168000 bytes 2023-12-04T14:26:14.361 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-12-04T14:26:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3600 IOs, 1719409664 bytes 2023-12-04T14:26:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3700 IOs, 1727807488 bytes 2023-12-04T14:26:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3800 IOs, 1783622656 bytes 2023-12-04T14:26:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:3900 IOs, 1806590976 bytes 2023-12-04T14:26:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4000 IOs, 1842804736 bytes 2023-12-04T14:26:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4100 IOs, 1908626432 bytes 2023-12-04T14:26:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4200 IOs, 1940933632 bytes 2023-12-04T14:26:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4300 IOs, 1941614592 bytes 2023-12-04T14:26:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4400 IOs, 2007821312 bytes 2023-12-04T14:26:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4500 IOs, 2040361984 bytes 2023-12-04T14:26:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4600 IOs, 2053388288 bytes 2023-12-04T14:26:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4700 IOs, 2140667904 bytes 2023-12-04T14:26:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4800 IOs, 2233728000 bytes 2023-12-04T14:26:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:4900 IOs, 2258511872 bytes 2023-12-04T14:26:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5000 IOs, 2270262272 bytes 2023-12-04T14:26:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5100 IOs, 2283052032 bytes 2023-12-04T14:26:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5200 IOs, 2343993344 bytes 2023-12-04T14:26:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5300 IOs, 2384041984 bytes 2023-12-04T14:26:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5400 IOs, 2431280128 bytes 2023-12-04T14:26:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5500 IOs, 2449336320 bytes 2023-12-04T14:26:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5600 IOs, 2515978240 bytes 2023-12-04T14:26:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5700 IOs, 2541765632 bytes 2023-12-04T14:26:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5800 IOs, 2561906688 bytes 2023-12-04T14:26:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:5900 IOs, 2583014400 bytes 2023-12-04T14:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6000 IOs, 2621221888 bytes 2023-12-04T14:26:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6100 IOs, 2679333888 bytes 2023-12-04T14:26:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6200 IOs, 2815192064 bytes 2023-12-04T14:26:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6300 IOs, 2852429824 bytes 2023-12-04T14:26:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6400 IOs, 2907767808 bytes 2023-12-04T14:26:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6500 IOs, 2952129536 bytes 2023-12-04T14:26:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6600 IOs, 3009038336 bytes 2023-12-04T14:26:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6700 IOs, 3046537216 bytes 2023-12-04T14:26:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6800 IOs, 3055113216 bytes 2023-12-04T14:26:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:6900 IOs, 3118123008 bytes 2023-12-04T14:26:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7000 IOs, 3155401728 bytes 2023-12-04T14:26:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7100 IOs, 3194435584 bytes 2023-12-04T14:26:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7200 IOs, 3238035456 bytes 2023-12-04T14:26:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7300 IOs, 3299224576 bytes 2023-12-04T14:26:19.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-12-04T14:26:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7400 IOs, 3383643136 bytes 2023-12-04T14:26:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7500 IOs, 3466656768 bytes 2023-12-04T14:26:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7600 IOs, 3524969472 bytes 2023-12-04T14:26:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7700 IOs, 3577123840 bytes 2023-12-04T14:26:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7800 IOs, 3634317312 bytes 2023-12-04T14:26:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:7900 IOs, 3675103232 bytes 2023-12-04T14:26:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8000 IOs, 3741274112 bytes 2023-12-04T14:26:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8100 IOs, 3807764480 bytes 2023-12-04T14:26:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8200 IOs, 3860819968 bytes 2023-12-04T14:26:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8300 IOs, 3891905536 bytes 2023-12-04T14:26:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8400 IOs, 3908752384 bytes 2023-12-04T14:26:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8500 IOs, 3947225088 bytes 2023-12-04T14:26:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8600 IOs, 4013934592 bytes 2023-12-04T14:26:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8700 IOs, 4044413952 bytes 2023-12-04T14:26:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8800 IOs, 4116918272 bytes 2023-12-04T14:26:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:8900 IOs, 4183023616 bytes 2023-12-04T14:26:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:9000 IOs, 4226799616 bytes 2023-12-04T14:26:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:9100 IOs, 4265963520 bytes 2023-12-04T14:26:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:9200 IOs, 4311361536 bytes 2023-12-04T14:26:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:9300 IOs, 4331796480 bytes 2023-12-04T14:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:9400 IOs, 4356050944 bytes 2023-12-04T14:26:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:9500 IOs, 4364252160 bytes 2023-12-04T14:26:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:9600 IOs, 4397194240 bytes 2023-12-04T14:26:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:9700 IOs, 4421665792 bytes 2023-12-04T14:26:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:9800 IOs, 4435875840 bytes 2023-12-04T14:26:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:9900 IOs, 4495610880 bytes 2023-12-04T14:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ error_code=124 2023-12-04T14:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set -e 2023-12-04T14:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 124 -eq 124 ']' 2023-12-04T14:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-12-04T14:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image_count=50 2023-12-04T14:26:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:26:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local count 2023-12-04T14:26:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local last_count=0 2023-12-04T14:26:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:26:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 1 40 2023-12-04T14:26:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in `seq 1 40` 2023-12-04T14:26:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 1 -ne 1 2023-12-04T14:26:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-04T14:26:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep 'images: ' 2023-12-04T14:26:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cut '-d ' -f 2 2023-12-04T14:26:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:26:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ count=50 2023-12-04T14:26:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 50 = 50 2023-12-04T14:26:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-12-04T14:26:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:26:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state 2023-12-04T14:26:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 1 40 2023-12-04T14:26:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in `seq 1 40` 2023-12-04T14:26:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 1 -ne 1 2023-12-04T14:26:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-04T14:26:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep 'image health:' 2023-12-04T14:26:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cut '-d ' -f 3 2023-12-04T14:26:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:26:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=OK 2023-12-04T14:26:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test OK = ERROR 2023-12-04T14:26:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test OK = OK 2023-12-04T14:26:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-12-04T14:26:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:26:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image_count=50 2023-12-04T14:26:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:26:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local count 2023-12-04T14:26:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local last_count=0 2023-12-04T14:26:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:26:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 1 40 2023-12-04T14:26:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in `seq 1 40` 2023-12-04T14:26:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 1 -ne 1 2023-12-04T14:26:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-04T14:26:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep 'images: ' 2023-12-04T14:26:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cut '-d ' -f 2 2023-12-04T14:26:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:26:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ count=50 2023-12-04T14:26:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 50 = 50 2023-12-04T14:26:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-12-04T14:26:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:26:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:26:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state 2023-12-04T14:26:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 1 40 2023-12-04T14:26:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in `seq 1 40` 2023-12-04T14:26:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 1 -ne 1 2023-12-04T14:26:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-04T14:26:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep 'image health:' 2023-12-04T14:26:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cut '-d ' -f 3 2023-12-04T14:26:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:26:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=WARNING 2023-12-04T14:26:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test WARNING = ERROR 2023-12-04T14:26:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test WARNING = OK 2023-12-04T14:26:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in `seq 1 40` 2023-12-04T14:26:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 2 -ne 1 2023-12-04T14:26:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 30 2023-12-04T14:26:25.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-12-04T14:26:30.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-12-04T14:26:35.315 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-04T14:26:35.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-04T14:26:35.316 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-04T14:26:35.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi176.stderr:2023-12-04T14:26:35.316+0000 7f3f07243640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 103666) UID: 1000 2023-12-04T14:26:35.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-12-04T14:26:41.381 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-12-04T14:26:44.317 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-04T14:26:44.317 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:26:44.317 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-04T14:26:44.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-04T14:26:44.318 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-04T14:26:44.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-04T14:26:44.321 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-04T14:26:46.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-12-04T14:26:52.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-12-04T14:26:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-04T14:26:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep 'image health:' 2023-12-04T14:26:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cut '-d ' -f 3 2023-12-04T14:26:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: failed to query services: (13) Permission denied 2023-12-04T14:26:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ state=OK 2023-12-04T14:26:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test OK = ERROR 2023-12-04T14:26:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test OK = OK 2023-12-04T14:26:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ testlog 'TEST: compare many images' 2023-12-04T14:26:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ tee -a /tmp/tmp.rDwoQpiP9p/rbd-mirror.test.log 2023-12-04T14:26:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ date '+%F %T' 2023-12-04T14:26:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04 14:26:53 TEST: compare many images 2023-12-04T14:26:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:2023-12-04 14:26:53 TEST: compare many images 2023-12-04T14:26:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 1 50 2023-12-04T14:26:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:26:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_1 2023-12-04T14:26:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_1 snap 2023-12-04T14:26:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:26:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:26:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-12-04T14:26:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:26:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-12-04T14:26:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:26:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:26:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:26:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:26:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:26:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-12-04T14:26:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:26:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:26:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:26:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:26:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-12-04T14:26:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:26:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:26:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:26:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:26:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:26:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:26:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:26:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:26:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:26:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:26:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:26:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-12-04T14:26:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:26:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:26:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:26:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:26:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:26:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:26:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:26:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:26:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:26:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:26:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:26:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:26:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:26:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:26:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:26:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:26:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:26:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:26:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:26:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:26:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:26:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:26:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:26:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:26:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:26:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:26:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:26:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:26:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:26:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:26:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:26:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:26:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:26:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-12-04T14:26:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2023-12-04T14:26:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:26:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:26:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:26:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2023-12-04T14:26:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:26:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:26:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:26:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:26:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:26:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:26:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:26:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:26:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:26:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:26:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:26:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:26:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:26:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:26:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:26:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:26:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:26:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:26:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:26:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:26:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:26:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:26:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:26:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:26:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:26:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:26:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:26:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:26:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:26:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:26:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:26:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:26:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:26:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2023-12-04T14:26:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2023-12-04T14:26:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:26:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:26:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:26:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2023-12-04T14:26:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:26:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:26:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:26:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:26:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:26:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:26:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:26:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:26:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:26:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:26:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:26:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:26:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:26:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:26:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:26:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:26:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:26:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:26:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:26:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:26:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:26:55.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:26:55.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:26:55.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:26:55.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2023-12-04T14:26:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2023-12-04T14:26:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:26:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:26:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:26:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_1 2023-12-04T14:26:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:26:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:26:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:26:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:26:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:26:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:26:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:26:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:26:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:26:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:26:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:26:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:26:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:26:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:26:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:26:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:26:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:26:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:26:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:26:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:26:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:26:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:26:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:26:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:26:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:26:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:26:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:26:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:26:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:26:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:26:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:26:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:26:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:26:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_1 2023-12-04T14:26:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:26:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:26:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:26:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:26:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:26:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:26:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:26:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-12-04T14:26:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:26:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:26:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:26:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-12-04T14:26:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:26:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_1.status 2023-12-04T14:26:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_1.status 2023-12-04T14:26:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:26:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_1.status 2023-12-04T14:26:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_1.status 2023-12-04T14:26:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_1.status 2023-12-04T14:26:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:26:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_1.status 2023-12-04T14:26:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-12-04T14:26:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-12-04T14:26:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 119 2023-12-04T14:26:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_1.status 2023-12-04T14:26:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_1.status 2023-12-04T14:26:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-12-04T14:26:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:26:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:26:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=119 2023-12-04T14:26:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:26:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:26:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:26:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_1.status 2023-12-04T14:26:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:26:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_1.status 2023-12-04T14:26:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-04T14:26:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:26:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:26:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:26:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:26:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:26:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:26:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:26:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 119 = '' 2023-12-04T14:26:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:26:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:26:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_1.status 2023-12-04T14:26:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:26:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_1.status 2023-12-04T14:26:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-04T14:26:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:26:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:26:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:26:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:26:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:26:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:26:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:26:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 119 = '' 2023-12-04T14:26:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:26:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:26:57.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-12-04T14:26:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_1.status 2023-12-04T14:26:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:26:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_1.status 2023-12-04T14:26:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-04T14:26:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:26:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:26:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=119 2023-12-04T14:26:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 119 = 119 2023-12-04T14:26:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-12-04T14:26:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:26:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:26:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:26:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:26:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:26:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-12-04T14:26:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_1': 2023-12-04T14:26:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:26:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:26:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:26:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 11a246f45543 2023-12-04T14:26:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.11a246f45543 2023-12-04T14:26:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:26:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:26:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:26:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:26:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:17:32 2023 2023-12-04T14:26:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:17:32 2023 2023-12-04T14:26:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:17:32 2023 2023-12-04T14:26:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:26:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:26:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:26:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: b928065e-ca9b-41a2-9e4f-db3d8351e631 2023-12-04T14:26:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:26:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_1 snap 2023-12-04T14:26:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_1 2023-12-04T14:26:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:26:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:26:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_1.export 2023-12-04T14:26:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_1.export 2023-12-04T14:26:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_1.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_1.export 2023-12-04T14:26:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_1.export 2023-12-04T14:26:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:26:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_1.export 2023-12-04T14:26:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:26:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_1.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_1.export 2023-12-04T14:26:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_1.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_1.export 2023-12-04T14:26:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:26:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:26:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_2 2023-12-04T14:26:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_2 snap 2023-12-04T14:26:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:26:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:26:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:26:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:26:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:27:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-12-04T14:27:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-12-04T14:27:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:27:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-12-04T14:27:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:27:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:27:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:27:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-12-04T14:27:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:27:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:27:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:27:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:27:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:27:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:27:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:27:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:27:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-04T14:27:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:27:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:27:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:27:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:27:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:27:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:27:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:27:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-04T14:27:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2023-12-04T14:27:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-12-04T14:27:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:27:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:27:01.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:27:01.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:27:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-12-04T14:27:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2023-12-04T14:27:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-12-04T14:27:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:27:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:27:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:27:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:27:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:27:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:27:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:27:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-12-04T14:27:01.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2023-12-04T14:27:01.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2023-12-04T14:27:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:27:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:27:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:27:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:27:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:27:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:27:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:27:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:27:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:27:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:27:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:27:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:27:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2023-12-04T14:27:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:27:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:27:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:27:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:27:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:27:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:27:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:27:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-12-04T14:27:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:27:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:27:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-12-04T14:27:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_2.status 2023-12-04T14:27:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_2.status 2023-12-04T14:27:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_2.status 2023-12-04T14:27:01.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_2.status 2023-12-04T14:27:01.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_2.status 2023-12-04T14:27:01.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.status 2023-12-04T14:27:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-12-04T14:27:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-12-04T14:27:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 121 2023-12-04T14:27:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_2.status 2023-12-04T14:27:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_2.status 2023-12-04T14:27:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-12-04T14:27:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:27:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=121 2023-12-04T14:27:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:27:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.status 2023-12-04T14:27:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.status 2023-12-04T14:27:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-04T14:27:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=69 2023-12-04T14:27:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 121 = 69 2023-12-04T14:27:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:27:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.status 2023-12-04T14:27:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.status 2023-12-04T14:27:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-04T14:27:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:02.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=69 2023-12-04T14:27:02.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 121 = 69 2023-12-04T14:27:02.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:27:03.006 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-12-04T14:27:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.status 2023-12-04T14:27:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.status 2023-12-04T14:27:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-04T14:27:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 121 = '' 2023-12-04T14:27:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:27:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.status 2023-12-04T14:27:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.status 2023-12-04T14:27:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-04T14:27:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=121 2023-12-04T14:27:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 121 = 121 2023-12-04T14:27:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-12-04T14:27:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:27:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-12-04T14:27:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_2': 2023-12-04T14:27:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:27:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:27:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:27:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 11a2e0a5aaf0 2023-12-04T14:27:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.11a2e0a5aaf0 2023-12-04T14:27:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:27:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:27:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:27:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:27:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:17:42 2023 2023-12-04T14:27:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:17:42 2023 2023-12-04T14:27:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:17:42 2023 2023-12-04T14:27:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:27:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:27:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:27:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 9fe30a7e-3975-4be8-9ddc-ffdbf8f787e1 2023-12-04T14:27:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:27:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_2 snap 2023-12-04T14:27:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_2 2023-12-04T14:27:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:27:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_2.export 2023-12-04T14:27:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.export 2023-12-04T14:27:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_2.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.export 2023-12-04T14:27:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_2.export 2023-12-04T14:27:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.export 2023-12-04T14:27:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_2.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.export 2023-12-04T14:27:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_2.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_2.export 2023-12-04T14:27:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:27:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_3 2023-12-04T14:27:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_3 snap 2023-12-04T14:27:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:27:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-12-04T14:27:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-12-04T14:27:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:27:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-12-04T14:27:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:27:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:27:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:27:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2023-12-04T14:27:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:27:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:27:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:27:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:27:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:27:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:27:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:27:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:27:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-12-04T14:27:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:27:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:27:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:27:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:27:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:27:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:27:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:27:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:27:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:27:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:27:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:27:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:27:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-12-04T14:27:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2023-12-04T14:27:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2023-12-04T14:27:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:27:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:27:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:27:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:27:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:27:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:27:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:27:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:27:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:27:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:27:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:27:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:27:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2023-12-04T14:27:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2023-12-04T14:27:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2023-12-04T14:27:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:27:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:27:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:27:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:27:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:27:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:27:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:27:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:27:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:27:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:27:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:27:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:27:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2023-12-04T14:27:08.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-12-04T14:27:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2023-12-04T14:27:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_3 2023-12-04T14:27:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:27:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:27:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:27:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:27:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:27:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:27:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:27:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:27:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:27:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:27:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:27:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:27:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_3 2023-12-04T14:27:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:27:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:27:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:27:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:27:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:27:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:27:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:27:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-12-04T14:27:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:27:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:27:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-12-04T14:27:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_3.status 2023-12-04T14:27:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_3.status 2023-12-04T14:27:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_3.status 2023-12-04T14:27:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_3.status 2023-12-04T14:27:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_3.status 2023-12-04T14:27:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.status 2023-12-04T14:27:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-12-04T14:27:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-12-04T14:27:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 123 2023-12-04T14:27:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_3.status 2023-12-04T14:27:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_3.status 2023-12-04T14:27:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-12-04T14:27:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:27:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=123 2023-12-04T14:27:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:27:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.status 2023-12-04T14:27:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.status 2023-12-04T14:27:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-04T14:27:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=70 2023-12-04T14:27:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 123 = 70 2023-12-04T14:27:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:27:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.status 2023-12-04T14:27:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.status 2023-12-04T14:27:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-04T14:27:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 123 = '' 2023-12-04T14:27:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:27:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.status 2023-12-04T14:27:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.status 2023-12-04T14:27:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-04T14:27:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 123 = '' 2023-12-04T14:27:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:27:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.status 2023-12-04T14:27:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.status 2023-12-04T14:27:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-04T14:27:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=123 2023-12-04T14:27:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 123 = 123 2023-12-04T14:27:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-12-04T14:27:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:27:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-12-04T14:27:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_3': 2023-12-04T14:27:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:27:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:27:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:27:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3a1951f0 2023-12-04T14:27:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3a1951f0 2023-12-04T14:27:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:27:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:27:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:27:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:27:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:17:54 2023 2023-12-04T14:27:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:17:54 2023 2023-12-04T14:27:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:17:54 2023 2023-12-04T14:27:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:27:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:27:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:27:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: d5429052-81ab-46d6-ab3d-16695051435d 2023-12-04T14:27:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:27:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_3 snap 2023-12-04T14:27:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_3 2023-12-04T14:27:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:27:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_3.export 2023-12-04T14:27:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.export 2023-12-04T14:27:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_3.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.export 2023-12-04T14:27:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_3.export 2023-12-04T14:27:13.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-12-04T14:27:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.export 2023-12-04T14:27:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_3.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.export 2023-12-04T14:27:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_3.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_3.export 2023-12-04T14:27:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:27:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_4 2023-12-04T14:27:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_4 snap 2023-12-04T14:27:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:27:14.740 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6605863527257584) > weight (0.25) 2023-12-04T14:27:14.740 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-04T14:27:14.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-04T14:27:14.741 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-04T14:27:14.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi176.stderr:2023-12-04T14:27:14.740+0000 7fe4dacc3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 101825) UID: 1000 2023-12-04T14:27:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-12-04T14:27:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-12-04T14:27:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:27:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-12-04T14:27:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:27:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:27:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:27:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-12-04T14:27:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:27:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:27:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:27:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:27:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:27:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:27:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:27:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:27:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-04T14:27:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:27:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:27:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-04T14:27:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-04T14:27:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-04T14:27:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:27:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:27:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:27:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:27:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:27:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:27:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:27:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:27:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-04T14:27:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-04T14:27:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-04T14:27:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:27:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:27:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:27:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:27:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:27:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:27:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:27:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-04T14:27:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-04T14:27:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:16.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:16.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-04T14:27:16.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:27:16.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:16.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:16.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:27:16.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:27:16.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:16.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:16.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:16.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:27:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:27:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:27:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-04T14:27:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:27:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:27:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:27:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:27:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:27:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:27:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:27:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-12-04T14:27:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:27:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:27:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-12-04T14:27:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_4.status 2023-12-04T14:27:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_4.status 2023-12-04T14:27:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_4.status 2023-12-04T14:27:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_4.status 2023-12-04T14:27:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_4.status 2023-12-04T14:27:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_4.status 2023-12-04T14:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-12-04T14:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-12-04T14:27:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 125 2023-12-04T14:27:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_4.status 2023-12-04T14:27:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_4.status 2023-12-04T14:27:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-12-04T14:27:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:27:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=125 2023-12-04T14:27:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:27:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_4.status 2023-12-04T14:27:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_4.status 2023-12-04T14:27:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-04T14:27:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 125 = '' 2023-12-04T14:27:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:27:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_4.status 2023-12-04T14:27:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_4.status 2023-12-04T14:27:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-04T14:27:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 125 = '' 2023-12-04T14:27:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:27:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_4.status 2023-12-04T14:27:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_4.status 2023-12-04T14:27:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-04T14:27:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=125 2023-12-04T14:27:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 125 = 125 2023-12-04T14:27:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-12-04T14:27:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:27:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:19.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-12-04T14:27:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-12-04T14:27:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_4': 2023-12-04T14:27:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:27:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:27:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:27:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3c3da3ce1 2023-12-04T14:27:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3c3da3ce1 2023-12-04T14:27:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:27:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:27:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:27:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:27:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:06 2023 2023-12-04T14:27:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:18:06 2023 2023-12-04T14:27:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:06 2023 2023-12-04T14:27:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:27:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:27:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:27:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 60498ab7-7310-4365-aa39-30c7019acf9f 2023-12-04T14:27:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:27:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_4 snap 2023-12-04T14:27:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_4 2023-12-04T14:27:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:27:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_4.export 2023-12-04T14:27:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_4.export 2023-12-04T14:27:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_4.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_4.export 2023-12-04T14:27:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_4.export 2023-12-04T14:27:19.742 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-04T14:27:19.742 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:27:19.743 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-04T14:27:19.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-04T14:27:19.743 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-04T14:27:19.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-04T14:27:19.748 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2023-12-04T14:27:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_4.export 2023-12-04T14:27:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_4.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_4.export 2023-12-04T14:27:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_4.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_4.export 2023-12-04T14:27:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:27:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_5 2023-12-04T14:27:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_5 snap 2023-12-04T14:27:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:27:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-12-04T14:27:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-12-04T14:27:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:27:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-12-04T14:27:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:27:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:27:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:27:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-12-04T14:27:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:27:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:27:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:27:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:27:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:27:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:27:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:27:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:27:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-12-04T14:27:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:27:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:27:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:27:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:27:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:27:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:27:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:27:22.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:22.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:22.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-12-04T14:27:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-04T14:27:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2023-12-04T14:27:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:27:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:27:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:27:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:27:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2023-12-04T14:27:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-04T14:27:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2023-12-04T14:27:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:27:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:27:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:27:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:27:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:27:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:27:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2023-12-04T14:27:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-04T14:27:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:27:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:27:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2023-12-04T14:27:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:27:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:27:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:27:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:27:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:27:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:27:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:27:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-12-04T14:27:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:27:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:27:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-12-04T14:27:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_5.status 2023-12-04T14:27:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_5.status 2023-12-04T14:27:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_5.status 2023-12-04T14:27:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_5.status 2023-12-04T14:27:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_5.status 2023-12-04T14:27:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.status 2023-12-04T14:27:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-12-04T14:27:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-12-04T14:27:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 127 2023-12-04T14:27:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_5.status 2023-12-04T14:27:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_5.status 2023-12-04T14:27:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-12-04T14:27:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:27:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=127 2023-12-04T14:27:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:27:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.status 2023-12-04T14:27:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.status 2023-12-04T14:27:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-04T14:27:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=72 2023-12-04T14:27:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 127 = 72 2023-12-04T14:27:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:27:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.status 2023-12-04T14:27:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.status 2023-12-04T14:27:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-04T14:27:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=72 2023-12-04T14:27:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 127 = 72 2023-12-04T14:27:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:27:24.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2023-12-04T14:27:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.status 2023-12-04T14:27:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.status 2023-12-04T14:27:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-04T14:27:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 127 = '' 2023-12-04T14:27:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:27:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.status 2023-12-04T14:27:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.status 2023-12-04T14:27:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-04T14:27:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:26.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=127 2023-12-04T14:27:26.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 127 = 127 2023-12-04T14:27:26.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-12-04T14:27:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:27:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-12-04T14:27:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_5': 2023-12-04T14:27:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:27:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d33c6af833 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d33c6af833 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:17 2023 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:18:17 2023 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:17 2023 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 90bc0c25-b4ac-4b4a-8f12-bbe6c286b2cd 2023-12-04T14:27:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:27:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_5 snap 2023-12-04T14:27:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_5 2023-12-04T14:27:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:27:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_5.export 2023-12-04T14:27:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.export 2023-12-04T14:27:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_5.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.export 2023-12-04T14:27:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_5.export 2023-12-04T14:27:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.export 2023-12-04T14:27:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_5.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.export 2023-12-04T14:27:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_5.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_5.export 2023-12-04T14:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_6 2023-12-04T14:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_6 snap 2023-12-04T14:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:27:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-12-04T14:27:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-12-04T14:27:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:27:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-12-04T14:27:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:27:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:27:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:27:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2023-12-04T14:27:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:27:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:27:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:27:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:27:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:27:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:27:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:27:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:27:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-12-04T14:27:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:27:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:27:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:27:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:27:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:27:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:27:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:27:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:27:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:27:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-12-04T14:27:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-12-04T14:27:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2023-12-04T14:27:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:27:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:27:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:27:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:27:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:27:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:27:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:27:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:27:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:27:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:27:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:27:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:27:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:29.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:29.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:29.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2023-12-04T14:27:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-12-04T14:27:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2023-12-04T14:27:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:27:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:27:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:27:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:27:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:27:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:27:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:27:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:27:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:27:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:27:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:27:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:27:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2023-12-04T14:27:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-12-04T14:27:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_6 2023-12-04T14:27:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:27:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:27:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:27:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:27:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:27:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:27:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_6 2023-12-04T14:27:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:27:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:27:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:27:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:27:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:27:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:27:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:27:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-12-04T14:27:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:27:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:27:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-12-04T14:27:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_6.status 2023-12-04T14:27:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_6.status 2023-12-04T14:27:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_6.status 2023-12-04T14:27:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_6.status 2023-12-04T14:27:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_6.status 2023-12-04T14:27:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.status 2023-12-04T14:27:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-12-04T14:27:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-12-04T14:27:30.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-12-04T14:27:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 129 2023-12-04T14:27:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_6.status 2023-12-04T14:27:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_6.status 2023-12-04T14:27:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-12-04T14:27:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:27:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=129 2023-12-04T14:27:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:27:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.status 2023-12-04T14:27:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.status 2023-12-04T14:27:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-04T14:27:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=73 2023-12-04T14:27:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 129 = 73 2023-12-04T14:27:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:27:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.status 2023-12-04T14:27:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.status 2023-12-04T14:27:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-04T14:27:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=73 2023-12-04T14:27:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 129 = 73 2023-12-04T14:27:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:27:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.status 2023-12-04T14:27:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.status 2023-12-04T14:27:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-04T14:27:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 129 = '' 2023-12-04T14:27:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:27:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.status 2023-12-04T14:27:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.status 2023-12-04T14:27:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-04T14:27:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=129 2023-12-04T14:27:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 129 = 129 2023-12-04T14:27:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-12-04T14:27:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:27:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-12-04T14:27:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_6': 2023-12-04T14:27:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:27:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:27:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:27:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3d00f5c86 2023-12-04T14:27:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3d00f5c86 2023-12-04T14:27:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:27:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:27:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:27:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:27:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:26 2023 2023-12-04T14:27:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:18:26 2023 2023-12-04T14:27:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:26 2023 2023-12-04T14:27:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:27:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:27:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:27:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 481d8ed8-0c1d-4f88-8ce7-bbcfd2363766 2023-12-04T14:27:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:27:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_6 snap 2023-12-04T14:27:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_6 2023-12-04T14:27:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:27:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_6.export 2023-12-04T14:27:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.export 2023-12-04T14:27:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_6.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.export 2023-12-04T14:27:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_6.export 2023-12-04T14:27:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.export 2023-12-04T14:27:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_6.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.export 2023-12-04T14:27:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_6.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_6.export 2023-12-04T14:27:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:27:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_7 2023-12-04T14:27:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_7 snap 2023-12-04T14:27:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:27:35.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-12-04T14:27:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-12-04T14:27:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-12-04T14:27:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:27:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-12-04T14:27:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:27:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:27:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:27:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-12-04T14:27:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:27:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:27:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:27:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:27:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:27:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:27:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:27:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:27:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-12-04T14:27:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:27:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:27:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:27:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:27:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:27:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-12-04T14:27:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-04T14:27:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:27:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:27:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:27:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:27:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:27:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:27:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:27:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:27:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2023-12-04T14:27:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-04T14:27:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2023-12-04T14:27:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:27:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:27:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:27:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:27:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:27:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:27:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:27:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:27:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:27:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:27:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:27:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2023-12-04T14:27:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-04T14:27:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2023-12-04T14:27:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:27:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:27:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:27:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:27:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:27:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:27:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:27:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:27:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:27:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:27:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:27:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:27:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2023-12-04T14:27:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:27:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:27:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:27:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:27:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:27:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:27:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:27:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-12-04T14:27:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:27:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:27:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-12-04T14:27:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_7.status 2023-12-04T14:27:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_7.status 2023-12-04T14:27:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_7.status 2023-12-04T14:27:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_7.status 2023-12-04T14:27:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_7.status 2023-12-04T14:27:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_7.status 2023-12-04T14:27:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-12-04T14:27:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-12-04T14:27:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 131 2023-12-04T14:27:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_7.status 2023-12-04T14:27:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_7.status 2023-12-04T14:27:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-12-04T14:27:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:27:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=131 2023-12-04T14:27:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:27:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_7.status 2023-12-04T14:27:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_7.status 2023-12-04T14:27:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-04T14:27:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 131 = '' 2023-12-04T14:27:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:27:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_7.status 2023-12-04T14:27:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_7.status 2023-12-04T14:27:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-04T14:27:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 131 = '' 2023-12-04T14:27:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:27:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_7.status 2023-12-04T14:27:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_7.status 2023-12-04T14:27:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-04T14:27:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=131 2023-12-04T14:27:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 131 = 131 2023-12-04T14:27:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-12-04T14:27:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:27:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-12-04T14:27:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_7': 2023-12-04T14:27:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:27:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:27:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:27:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3d23474f 2023-12-04T14:27:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3d23474f 2023-12-04T14:27:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:27:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:27:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:27:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:27:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:36 2023 2023-12-04T14:27:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:18:36 2023 2023-12-04T14:27:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:36 2023 2023-12-04T14:27:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:27:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:27:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:27:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 431573ac-8014-4867-b7be-738ee54b8f15 2023-12-04T14:27:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:27:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_7 snap 2023-12-04T14:27:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_7 2023-12-04T14:27:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:27:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_7.export 2023-12-04T14:27:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_7.export 2023-12-04T14:27:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_7.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_7.export 2023-12-04T14:27:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_7.export 2023-12-04T14:27:40.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-12-04T14:27:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_7.export 2023-12-04T14:27:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_7.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_7.export 2023-12-04T14:27:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_7.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_7.export 2023-12-04T14:27:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:27:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_8 2023-12-04T14:27:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_8 snap 2023-12-04T14:27:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:27:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-12-04T14:27:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-12-04T14:27:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:27:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-12-04T14:27:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:27:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:27:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:27:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2023-12-04T14:27:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:27:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:27:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:27:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:27:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:27:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:27:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:27:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:27:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-12-04T14:27:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:27:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:27:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:27:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:27:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:27:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:27:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-12-04T14:27:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2023-12-04T14:27:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2023-12-04T14:27:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:27:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2023-12-04T14:27:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2023-12-04T14:27:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2023-12-04T14:27:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:27:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:27:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:27:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:27:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2023-12-04T14:27:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2023-12-04T14:27:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_8 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:27:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:27:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:27:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_8 2023-12-04T14:27:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:27:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:27:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:27:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:27:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:27:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:27:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:27:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-12-04T14:27:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:27:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:27:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-12-04T14:27:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_8.status 2023-12-04T14:27:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_8.status 2023-12-04T14:27:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_8.status 2023-12-04T14:27:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_8.status 2023-12-04T14:27:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_8.status 2023-12-04T14:27:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.status 2023-12-04T14:27:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-12-04T14:27:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-12-04T14:27:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 133 2023-12-04T14:27:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_8.status 2023-12-04T14:27:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_8.status 2023-12-04T14:27:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-12-04T14:27:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:27:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=133 2023-12-04T14:27:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:27:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.status 2023-12-04T14:27:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.status 2023-12-04T14:27:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-04T14:27:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=75 2023-12-04T14:27:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 133 = 75 2023-12-04T14:27:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:27:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.status 2023-12-04T14:27:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.status 2023-12-04T14:27:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-04T14:27:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=75 2023-12-04T14:27:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 133 = 75 2023-12-04T14:27:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.status 2023-12-04T14:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.status 2023-12-04T14:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-04T14:27:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:45.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 133 = '' 2023-12-04T14:27:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:27:46.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-12-04T14:27:46.414 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:27:46.412+0000 7f4bfa770640 -1 reset not still connected to 0x56233d93e820 2023-12-04T14:27:46.414 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:27:46.412+0000 7f4bfa770640 -1 reset not still connected to 0x5623539a7520 2023-12-04T14:27:46.416 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:27:46.414+0000 7f4bfa770640 -1 reset not still connected to 0x562354245d40 2023-12-04T14:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.status 2023-12-04T14:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.status 2023-12-04T14:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-04T14:27:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=133 2023-12-04T14:27:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 133 = 133 2023-12-04T14:27:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-12-04T14:27:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:27:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-12-04T14:27:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_8': 2023-12-04T14:27:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:27:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:27:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:27:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3ad137cc4 2023-12-04T14:27:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3ad137cc4 2023-12-04T14:27:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:27:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:27:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:27:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:27:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:42 2023 2023-12-04T14:27:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:18:42 2023 2023-12-04T14:27:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:42 2023 2023-12-04T14:27:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:27:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:27:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:27:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 14168f82-d9d2-41fc-99ed-f1205d8ef156 2023-12-04T14:27:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:27:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_8 snap 2023-12-04T14:27:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_8 2023-12-04T14:27:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:27:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_8.export 2023-12-04T14:27:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.export 2023-12-04T14:27:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_8.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.export 2023-12-04T14:27:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_8.export 2023-12-04T14:27:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.export 2023-12-04T14:27:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_8.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.export 2023-12-04T14:27:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_8.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_8.export 2023-12-04T14:27:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:27:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_9 2023-12-04T14:27:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_9 snap 2023-12-04T14:27:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:27:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-12-04T14:27:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-12-04T14:27:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:27:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:27:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:27:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:27:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:27:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-12-04T14:27:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:27:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:27:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:27:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:27:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:27:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:27:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-12-04T14:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-04T14:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:27:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:27:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:27:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2023-12-04T14:27:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-04T14:27:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2023-12-04T14:27:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:27:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:27:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:27:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:27:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:27:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:27:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2023-12-04T14:27:51.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-04T14:27:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_9 2023-12-04T14:27:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:27:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:27:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:27:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:27:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:27:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:27:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:27:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_9 2023-12-04T14:27:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:27:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:27:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:27:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:27:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:27:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:27:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:27:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-12-04T14:27:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:27:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:27:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-12-04T14:27:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_9.status 2023-12-04T14:27:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_9.status 2023-12-04T14:27:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_9.status 2023-12-04T14:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_9.status 2023-12-04T14:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_9.status 2023-12-04T14:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_9.status 2023-12-04T14:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-12-04T14:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-12-04T14:27:51.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2023-12-04T14:27:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 135 2023-12-04T14:27:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_9.status 2023-12-04T14:27:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_9.status 2023-12-04T14:27:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-12-04T14:27:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:27:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=135 2023-12-04T14:27:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:27:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_9.status 2023-12-04T14:27:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_9.status 2023-12-04T14:27:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-04T14:27:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 135 = '' 2023-12-04T14:27:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:27:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_9.status 2023-12-04T14:27:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_9.status 2023-12-04T14:27:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-04T14:27:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 135 = '' 2023-12-04T14:27:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:27:53.421 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.418+0000 7f0527feb640 -1 reset not still connected to 0x563436cf6000 2023-12-04T14:27:53.421 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.418+0000 7f0527feb640 -1 reset not still connected to 0x5634371b25b0 2023-12-04T14:27:53.422 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.418+0000 7f0527feb640 -1 reset not still connected to 0x563437547e10 2023-12-04T14:27:53.422 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.418+0000 7f0527feb640 -1 reset not still connected to 0x56343b2836c0 2023-12-04T14:27:53.422 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.418+0000 7f0527feb640 -1 reset not still connected to 0x56343db2dc70 2023-12-04T14:27:53.422 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.418+0000 7f0527feb640 -1 reset not still connected to 0x56343e86d930 2023-12-04T14:27:53.422 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.418+0000 7f0527feb640 -1 reset not still connected to 0x56343f9a9930 2023-12-04T14:27:53.422 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.418+0000 7f0527feb640 -1 reset not still connected to 0x56343fa940d0 2023-12-04T14:27:53.422 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.418+0000 7f0527feb640 -1 reset not still connected to 0x563441f55ad0 2023-12-04T14:27:53.422 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.419+0000 7f0527feb640 -1 reset not still connected to 0x56344800b380 2023-12-04T14:27:53.422 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.419+0000 7f0527feb640 -1 reset not still connected to 0x56344946d520 2023-12-04T14:27:53.427 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.425+0000 7f0527feb640 -1 reset not still connected to 0x56344b43da00 2023-12-04T14:27:53.427 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.425+0000 7f0527feb640 -1 reset not still connected to 0x563451553930 2023-12-04T14:27:53.427 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.425+0000 7f0527feb640 -1 reset not still connected to 0x563451e171e0 2023-12-04T14:27:53.428 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.425+0000 7f0527feb640 -1 reset not still connected to 0x5634536561a0 2023-12-04T14:27:53.429 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.427+0000 7f0527feb640 -1 reset not still connected to 0x563456f651e0 2023-12-04T14:27:53.429 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.427+0000 7f0527feb640 -1 reset not still connected to 0x56345c40cdd0 2023-12-04T14:27:53.429 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.427+0000 7f0527feb640 -1 reset not still connected to 0x5634634df6c0 2023-12-04T14:27:53.429 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.427+0000 7f0527feb640 -1 reset not still connected to 0x563475cbc9c0 2023-12-04T14:27:53.430 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:27:53.427+0000 7f0527feb640 -1 reset not still connected to 0x56347ae2e820 2023-12-04T14:27:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_9.status 2023-12-04T14:27:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_9.status 2023-12-04T14:27:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-04T14:27:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=135 2023-12-04T14:27:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 135 = 135 2023-12-04T14:27:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-12-04T14:27:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:27:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-12-04T14:27:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_9': 2023-12-04T14:27:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:27:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:27:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:27:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3857e5ce3 2023-12-04T14:27:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3857e5ce3 2023-12-04T14:27:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:27:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:27:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:27:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:27:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:52 2023 2023-12-04T14:27:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:18:52 2023 2023-12-04T14:27:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:52 2023 2023-12-04T14:27:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:27:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:27:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:27:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: fc8c6534-5c92-48ae-b43e-099c17bc6417 2023-12-04T14:27:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:27:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_9 snap 2023-12-04T14:27:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_9 2023-12-04T14:27:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:27:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_9.export 2023-12-04T14:27:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_9.export 2023-12-04T14:27:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_9.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_9.export 2023-12-04T14:27:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_9.export 2023-12-04T14:27:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_9.export 2023-12-04T14:27:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:27:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_9.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_9.export 2023-12-04T14:27:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_9.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_9.export 2023-12-04T14:27:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:27:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_10 2023-12-04T14:27:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_10 snap 2023-12-04T14:27:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:27:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:27:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-12-04T14:27:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:27:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-12-04T14:27:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:27:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:27:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:27:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-12-04T14:27:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:27:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:27:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:27:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:27:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2023-12-04T14:27:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:27:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:27:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:27:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:27:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:27:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:27:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:27:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:27:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:27:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:27:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:27:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:27:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:27:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:27:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-12-04T14:27:57.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-12-04T14:27:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-04T14:27:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-12-04T14:27:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:27:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:27:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:27:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:27:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:27:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:27:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:27:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:27:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:27:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-12-04T14:27:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-04T14:27:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2023-12-04T14:27:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:27:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:27:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:27:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:27:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2023-12-04T14:27:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-04T14:27:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:27:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:27:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:27:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2023-12-04T14:27:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:27:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:27:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:27:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:27:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:27:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:27:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:27:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:27:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:27:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:27:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:27:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:27:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:27:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:27:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:27:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:27:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:27:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2023-12-04T14:27:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:27:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:27:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:27:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:27:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:27:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:27:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:27:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-12-04T14:27:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:27:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:27:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:27:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-12-04T14:27:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:27:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:27:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_10.status 2023-12-04T14:27:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_10.status 2023-12-04T14:27:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_10.status 2023-12-04T14:27:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_10.status 2023-12-04T14:27:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_10.status 2023-12-04T14:27:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:27:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_10.status 2023-12-04T14:27:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-12-04T14:27:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:27:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-12-04T14:27:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 137 2023-12-04T14:27:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_10.status 2023-12-04T14:27:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:27:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:27:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_10.status 2023-12-04T14:27:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-12-04T14:27:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:27:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=137 2023-12-04T14:27:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:27:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_10.status 2023-12-04T14:27:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:27:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_10.status 2023-12-04T14:27:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-04T14:27:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 137 = '' 2023-12-04T14:27:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:27:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_10.status 2023-12-04T14:27:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:27:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_10.status 2023-12-04T14:27:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-04T14:27:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:27:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:27:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:27:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:27:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:27:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 137 = '' 2023-12-04T14:27:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:27:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:27:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_10.status 2023-12-04T14:27:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:27:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_10.status 2023-12-04T14:27:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-04T14:27:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:27:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=137 2023-12-04T14:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 137 = 137 2023-12-04T14:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:27:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-12-04T14:27:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:27:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:27:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:27:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:27:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:27:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:27:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-12-04T14:28:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_10': 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3fd89eee1 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3fd89eee1 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:06 2023 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:19:06 2023 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:06 2023 2023-12-04T14:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 0d9ffbfe-5f0b-4261-82aa-d3c625dda130 2023-12-04T14:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:28:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_10 snap 2023-12-04T14:28:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_10 2023-12-04T14:28:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:28:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_10.export 2023-12-04T14:28:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_10.export 2023-12-04T14:28:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_10.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_10.export 2023-12-04T14:28:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_10.export 2023-12-04T14:28:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_10.export 2023-12-04T14:28:01.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_10.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_10.export 2023-12-04T14:28:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_10.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_10.export 2023-12-04T14:28:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:28:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_11 2023-12-04T14:28:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_11 snap 2023-12-04T14:28:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:28:01.322 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9547454147372003) > weight (0.25) 2023-12-04T14:28:01.322 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4303018946904492) > weight (0.25) 2023-12-04T14:28:01.322 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-04T14:28:01.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-04T14:28:01.323 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-04T14:28:01.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi176.stderr:2023-12-04T14:28:01.322+0000 7fd1ec997640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 100871) UID: 1000 2023-12-04T14:28:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-12-04T14:28:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-12-04T14:28:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:28:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:02.324 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-04T14:28:02.324 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:28:02.324 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-04T14:28:02.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-04T14:28:02.325 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-04T14:28:02.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-04T14:28:02.328 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2023-12-04T14:28:02.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-12-04T14:28:03.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-12-04T14:28:03.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:03.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:03.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:03.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:28:03.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:28:03.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:28:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2023-12-04T14:28:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:28:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:28:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:28:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:28:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:28:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:28:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:28:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:28:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-12-04T14:28:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:28:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:28:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:28:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:28:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:28:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:28:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-12-04T14:28:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-04T14:28:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:28:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:28:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:28:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2023-12-04T14:28:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-04T14:28:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:28:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:28:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2023-12-04T14:28:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-04T14:28:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_11 2023-12-04T14:28:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:28:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:28:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:28:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:28:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:28:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:28:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_11 2023-12-04T14:28:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:28:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:28:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:28:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:28:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:28:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_11.status 2023-12-04T14:28:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_11.status 2023-12-04T14:28:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_11.status 2023-12-04T14:28:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_11.status 2023-12-04T14:28:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_11.status 2023-12-04T14:28:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_11.status 2023-12-04T14:28:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-12-04T14:28:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-12-04T14:28:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 139 2023-12-04T14:28:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_11.status 2023-12-04T14:28:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_11.status 2023-12-04T14:28:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-12-04T14:28:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:28:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=139 2023-12-04T14:28:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:28:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_11.status 2023-12-04T14:28:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_11.status 2023-12-04T14:28:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-04T14:28:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 139 = '' 2023-12-04T14:28:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:28:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_11.status 2023-12-04T14:28:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_11.status 2023-12-04T14:28:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-04T14:28:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 139 = '' 2023-12-04T14:28:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:28:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_11.status 2023-12-04T14:28:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_11.status 2023-12-04T14:28:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-04T14:28:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=139 2023-12-04T14:28:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 139 = 139 2023-12-04T14:28:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-12-04T14:28:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:28:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-12-04T14:28:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_11': 2023-12-04T14:28:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:28:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:28:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:28:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d32b04c6bd 2023-12-04T14:28:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d32b04c6bd 2023-12-04T14:28:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:28:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:28:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:28:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:28:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:15 2023 2023-12-04T14:28:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:19:15 2023 2023-12-04T14:28:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:15 2023 2023-12-04T14:28:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:28:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:28:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:28:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 99047729-2de2-4c91-90dc-1e2bfe4f2b5f 2023-12-04T14:28:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:28:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_11 snap 2023-12-04T14:28:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_11 2023-12-04T14:28:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:28:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_11.export 2023-12-04T14:28:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_11.export 2023-12-04T14:28:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_11.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_11.export 2023-12-04T14:28:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_11.export 2023-12-04T14:28:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_11.export 2023-12-04T14:28:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_11.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_11.export 2023-12-04T14:28:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_11.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_11.export 2023-12-04T14:28:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:28:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_12 2023-12-04T14:28:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_12 snap 2023-12-04T14:28:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:28:07.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-12-04T14:28:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-12-04T14:28:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-12-04T14:28:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:28:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-12-04T14:28:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:28:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:28:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:28:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2023-12-04T14:28:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:28:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:28:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:28:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:28:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:28:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:28:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:28:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:28:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-04T14:28:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:28:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:28:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:28:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:28:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:28:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:28:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:28:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:28:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:28:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:28:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:28:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:28:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-04T14:28:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-04T14:28:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-04T14:28:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:28:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:28:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:28:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:28:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:28:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:28:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:28:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:28:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:28:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:28:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:28:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:28:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-04T14:28:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-04T14:28:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2023-12-04T14:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:28:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:28:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:28:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:28:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:28:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:28:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2023-12-04T14:28:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-04T14:28:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2023-12-04T14:28:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:28:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:28:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:28:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:28:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:28:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:28:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:28:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:28:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:28:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2023-12-04T14:28:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:28:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:28:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:28:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:28:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:28:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:28:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:28:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-12-04T14:28:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:28:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:28:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-12-04T14:28:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_12.status 2023-12-04T14:28:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_12.status 2023-12-04T14:28:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_12.status 2023-12-04T14:28:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_12.status 2023-12-04T14:28:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_12.status 2023-12-04T14:28:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:09.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_12.status 2023-12-04T14:28:09.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-12-04T14:28:09.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:09.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:09.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:09.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-12-04T14:28:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 141 2023-12-04T14:28:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_12.status 2023-12-04T14:28:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_12.status 2023-12-04T14:28:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-12-04T14:28:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:28:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=141 2023-12-04T14:28:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:28:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_12.status 2023-12-04T14:28:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_12.status 2023-12-04T14:28:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-04T14:28:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 141 = '' 2023-12-04T14:28:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:28:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_12.status 2023-12-04T14:28:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_12.status 2023-12-04T14:28:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-04T14:28:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 141 = '' 2023-12-04T14:28:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:28:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_12.status 2023-12-04T14:28:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_12.status 2023-12-04T14:28:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-04T14:28:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=141 2023-12-04T14:28:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 141 = 141 2023-12-04T14:28:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-12-04T14:28:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:28:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-12-04T14:28:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_12': 2023-12-04T14:28:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:28:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:28:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:28:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3719168a4 2023-12-04T14:28:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3719168a4 2023-12-04T14:28:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:28:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:28:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:28:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:28:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:25 2023 2023-12-04T14:28:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:19:25 2023 2023-12-04T14:28:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:25 2023 2023-12-04T14:28:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:28:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:28:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:28:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 5f322e9d-b400-4d44-81e0-ffdd216398b2 2023-12-04T14:28:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:28:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_12 snap 2023-12-04T14:28:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_12 2023-12-04T14:28:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:28:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_12.export 2023-12-04T14:28:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_12.export 2023-12-04T14:28:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_12.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_12.export 2023-12-04T14:28:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_12.export 2023-12-04T14:28:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_12.export 2023-12-04T14:28:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_12.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_12.export 2023-12-04T14:28:13.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-12-04T14:28:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_12.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_12.export 2023-12-04T14:28:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:28:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_13 2023-12-04T14:28:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_13 snap 2023-12-04T14:28:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:28:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-12-04T14:28:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-12-04T14:28:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:28:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-12-04T14:28:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:28:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:28:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:28:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-12-04T14:28:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:28:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:28:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:28:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:28:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:28:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:28:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:28:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:28:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-12-04T14:28:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:28:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:28:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:28:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:28:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:28:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:28:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:28:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:28:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:28:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:28:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:28:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:28:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-12-04T14:28:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-12-04T14:28:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2023-12-04T14:28:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:28:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:28:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:28:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:28:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:28:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:28:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:28:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:28:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:28:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:28:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:28:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:28:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2023-12-04T14:28:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-12-04T14:28:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2023-12-04T14:28:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:28:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:28:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:28:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:28:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:28:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:28:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:28:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:28:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:28:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:28:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:28:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:28:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2023-12-04T14:28:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-12-04T14:28:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_13 2023-12-04T14:28:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:28:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:28:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:28:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:28:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:28:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:28:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:28:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:28:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:28:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:28:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:28:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:28:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_13 2023-12-04T14:28:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:28:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:28:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:28:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:28:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:28:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:28:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:28:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-12-04T14:28:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-12-04T14:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_13.status 2023-12-04T14:28:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_13.status 2023-12-04T14:28:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_13.status 2023-12-04T14:28:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_13.status 2023-12-04T14:28:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_13.status 2023-12-04T14:28:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_13.status 2023-12-04T14:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-12-04T14:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-12-04T14:28:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 143 2023-12-04T14:28:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_13.status 2023-12-04T14:28:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_13.status 2023-12-04T14:28:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-12-04T14:28:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:28:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=143 2023-12-04T14:28:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:28:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_13.status 2023-12-04T14:28:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_13.status 2023-12-04T14:28:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-04T14:28:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 143 = '' 2023-12-04T14:28:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:28:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_13.status 2023-12-04T14:28:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_13.status 2023-12-04T14:28:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-04T14:28:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 143 = '' 2023-12-04T14:28:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:28:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_13.status 2023-12-04T14:28:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_13.status 2023-12-04T14:28:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-04T14:28:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=143 2023-12-04T14:28:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 143 = 143 2023-12-04T14:28:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-12-04T14:28:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:28:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:18.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2023-12-04T14:28:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-12-04T14:28:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_13': 2023-12-04T14:28:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:28:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:28:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:28:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d34174832a 2023-12-04T14:28:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d34174832a 2023-12-04T14:28:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:28:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:28:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:28:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:28:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:37 2023 2023-12-04T14:28:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:19:37 2023 2023-12-04T14:28:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:37 2023 2023-12-04T14:28:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:28:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:28:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:28:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: ac478d96-184c-4eaa-8c62-f6f0e8288dff 2023-12-04T14:28:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:28:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_13 snap 2023-12-04T14:28:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_13 2023-12-04T14:28:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:28:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_13.export 2023-12-04T14:28:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_13.export 2023-12-04T14:28:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_13.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_13.export 2023-12-04T14:28:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_13.export 2023-12-04T14:28:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_13.export 2023-12-04T14:28:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_13.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_13.export 2023-12-04T14:28:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_13.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_13.export 2023-12-04T14:28:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:28:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_14 2023-12-04T14:28:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_14 snap 2023-12-04T14:28:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:28:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-12-04T14:28:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-12-04T14:28:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:28:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-12-04T14:28:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:28:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:28:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:28:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-12-04T14:28:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:28:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:28:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:28:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:28:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:28:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:28:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:28:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:28:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-12-04T14:28:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:28:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:28:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:28:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:28:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:28:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:28:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:28:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-12-04T14:28:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-04T14:28:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2023-12-04T14:28:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:28:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:28:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:28:21.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:21.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:21.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:21.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2023-12-04T14:28:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-04T14:28:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2023-12-04T14:28:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:28:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:28:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:28:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:28:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:28:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:28:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:28:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2023-12-04T14:28:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-04T14:28:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_14 2023-12-04T14:28:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:28:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:28:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:28:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:28:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:28:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:28:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:28:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:28:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:28:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:28:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:28:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:28:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_14 2023-12-04T14:28:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:28:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:28:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:28:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:28:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:28:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:28:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:28:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-12-04T14:28:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:28:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:28:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-12-04T14:28:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_14.status 2023-12-04T14:28:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_14.status 2023-12-04T14:28:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_14.status 2023-12-04T14:28:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_14.status 2023-12-04T14:28:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_14.status 2023-12-04T14:28:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.status 2023-12-04T14:28:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-12-04T14:28:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-12-04T14:28:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 145 2023-12-04T14:28:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_14.status 2023-12-04T14:28:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_14.status 2023-12-04T14:28:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-12-04T14:28:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:28:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=145 2023-12-04T14:28:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:28:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.status 2023-12-04T14:28:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.status 2023-12-04T14:28:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-04T14:28:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=81 2023-12-04T14:28:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 145 = 81 2023-12-04T14:28:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:28:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.status 2023-12-04T14:28:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.status 2023-12-04T14:28:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-04T14:28:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=81 2023-12-04T14:28:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 145 = 81 2023-12-04T14:28:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:28:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.status 2023-12-04T14:28:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.status 2023-12-04T14:28:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-04T14:28:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 145 = '' 2023-12-04T14:28:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:28:24.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-12-04T14:28:24.269 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:28:24.268+0000 7f0527feb640 -1 reset not still connected to 0x56347ae2e820 2023-12-04T14:28:24.274 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:28:24.274+0000 7f0527feb640 -1 reset not still connected to 0x56347ec1cea0 2023-12-04T14:28:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.status 2023-12-04T14:28:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.status 2023-12-04T14:28:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-04T14:28:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=145 2023-12-04T14:28:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 145 = 145 2023-12-04T14:28:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-12-04T14:28:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:28:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-12-04T14:28:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_14': 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3f464ab05 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3f464ab05 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:45 2023 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:19:45 2023 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:45 2023 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:28:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:28:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:28:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 8794a2f7-791e-4d63-afdb-a1f297b316fa 2023-12-04T14:28:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:28:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_14 snap 2023-12-04T14:28:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_14 2023-12-04T14:28:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:28:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_14.export 2023-12-04T14:28:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.export 2023-12-04T14:28:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_14.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.export 2023-12-04T14:28:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_14.export 2023-12-04T14:28:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.export 2023-12-04T14:28:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_14.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.export 2023-12-04T14:28:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_14.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_14.export 2023-12-04T14:28:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:28:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_15 2023-12-04T14:28:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_15 snap 2023-12-04T14:28:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:28:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-12-04T14:28:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-12-04T14:28:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:28:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-12-04T14:28:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:28:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:28:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:28:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-12-04T14:28:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:28:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:28:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:28:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:28:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:28:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:28:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:28:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:28:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-12-04T14:28:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:28:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:28:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:28:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:28:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:28:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:28:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-12-04T14:28:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-12-04T14:28:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:28:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:28:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:28:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2023-12-04T14:28:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-12-04T14:28:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2023-12-04T14:28:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:28:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2023-12-04T14:28:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-12-04T14:28:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_15 2023-12-04T14:28:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:28:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:28:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:28:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:28:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:28:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:28:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_15 2023-12-04T14:28:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:28:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:28:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:28:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:28:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_15.status 2023-12-04T14:28:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_15.status 2023-12-04T14:28:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_15.status 2023-12-04T14:28:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_15.status 2023-12-04T14:28:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_15.status 2023-12-04T14:28:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.status 2023-12-04T14:28:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-12-04T14:28:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-12-04T14:28:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 147 2023-12-04T14:28:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_15.status 2023-12-04T14:28:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_15.status 2023-12-04T14:28:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-12-04T14:28:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:28:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=147 2023-12-04T14:28:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:28:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.status 2023-12-04T14:28:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.status 2023-12-04T14:28:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-04T14:28:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:29.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-12-04T14:28:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=82 2023-12-04T14:28:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 147 = 82 2023-12-04T14:28:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:28:29.749 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5281650282976375) > weight (0.25) 2023-12-04T14:28:29.749 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8095101291785064) > weight (0.25) 2023-12-04T14:28:29.749 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7659560619445008) > weight (0.25) 2023-12-04T14:28:29.749 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.40207591551996413) > weight (0.25) 2023-12-04T14:28:29.749 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-12-04T14:28:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.status 2023-12-04T14:28:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.status 2023-12-04T14:28:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-04T14:28:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=82 2023-12-04T14:28:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 147 = 82 2023-12-04T14:28:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:28:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.status 2023-12-04T14:28:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.status 2023-12-04T14:28:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-04T14:28:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 147 = '' 2023-12-04T14:28:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:28:32.130 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.127+0000 7f78b9196640 -1 reset not still connected to 0x556ae8aa5380 2023-12-04T14:28:32.130 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.127+0000 7f78b9196640 -1 reset not still connected to 0x556ae8aa5930 2023-12-04T14:28:32.130 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.127+0000 7f78b9196640 -1 reset not still connected to 0x556ae8c48d00 2023-12-04T14:28:32.130 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.127+0000 7f78b9196640 -1 reset not still connected to 0x556ae8c49110 2023-12-04T14:28:32.130 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.127+0000 7f78b9196640 -1 reset not still connected to 0x556ae8c49c70 2023-12-04T14:28:32.130 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.128+0000 7f78b9196640 -1 reset not still connected to 0x556aea196750 2023-12-04T14:28:32.130 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.128+0000 7f78b9196640 -1 reset not still connected to 0x556aeaf45860 2023-12-04T14:28:32.130 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.128+0000 7f78b9196640 -1 reset not still connected to 0x556aeaf46410 2023-12-04T14:28:32.130 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.128+0000 7f78b9196640 -1 reset not still connected to 0x556aeaf465b0 2023-12-04T14:28:32.130 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.128+0000 7f78b9196640 -1 reset not still connected to 0x556aebae3c70 2023-12-04T14:28:32.130 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.128+0000 7f78b9196640 -1 reset not still connected to 0x556aebd0aea0 2023-12-04T14:28:32.131 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.128+0000 7f78b9196640 -1 reset not still connected to 0x556aec844270 2023-12-04T14:28:32.131 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.128+0000 7f78b9196640 -1 reset not still connected to 0x556aec943520 2023-12-04T14:28:32.131 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:28:32.128+0000 7f78b9196640 -1 reset not still connected to 0x556aec9436c0 2023-12-04T14:28:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.status 2023-12-04T14:28:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.status 2023-12-04T14:28:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-04T14:28:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=147 2023-12-04T14:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 147 = 147 2023-12-04T14:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-12-04T14:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:28:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-12-04T14:28:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_15': 2023-12-04T14:28:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:28:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:28:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:28:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3b3ad0417 2023-12-04T14:28:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3b3ad0417 2023-12-04T14:28:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:28:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:28:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:28:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:28:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:00 2023 2023-12-04T14:28:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:20:00 2023 2023-12-04T14:28:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:00 2023 2023-12-04T14:28:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:28:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:28:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:28:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 0e39536e-cf2c-4b79-ae6d-a5d1ede7920c 2023-12-04T14:28:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:28:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_15 snap 2023-12-04T14:28:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_15 2023-12-04T14:28:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:28:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_15.export 2023-12-04T14:28:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.export 2023-12-04T14:28:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_15.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.export 2023-12-04T14:28:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_15.export 2023-12-04T14:28:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.export 2023-12-04T14:28:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_15.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.export 2023-12-04T14:28:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_15.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_15.export 2023-12-04T14:28:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:28:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_16 2023-12-04T14:28:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_16 snap 2023-12-04T14:28:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:28:34.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-12-04T14:28:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-12-04T14:28:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-12-04T14:28:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:28:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-12-04T14:28:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:28:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:28:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:28:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-12-04T14:28:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:28:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:28:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:28:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:28:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:28:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:28:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:28:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:28:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-12-04T14:28:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:28:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:28:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:28:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:28:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:28:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:28:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:28:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:28:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:28:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:28:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:28:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:28:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-12-04T14:28:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-04T14:28:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2023-12-04T14:28:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:28:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:28:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:28:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:28:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:28:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:28:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:28:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:28:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:28:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:28:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:28:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:28:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2023-12-04T14:28:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-04T14:28:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2023-12-04T14:28:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:28:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:28:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:28:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:28:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:28:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:28:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:28:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:28:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:28:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:28:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2023-12-04T14:28:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-04T14:28:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2023-12-04T14:28:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:28:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:28:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:28:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:28:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:28:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:28:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:28:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:28:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:28:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:28:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:28:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:28:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2023-12-04T14:28:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:28:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:28:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:28:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:28:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:28:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:28:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:28:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-12-04T14:28:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:28:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:28:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-12-04T14:28:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_16.status 2023-12-04T14:28:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_16.status 2023-12-04T14:28:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_16.status 2023-12-04T14:28:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_16.status 2023-12-04T14:28:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_16.status 2023-12-04T14:28:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_16.status 2023-12-04T14:28:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-12-04T14:28:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-12-04T14:28:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 149 2023-12-04T14:28:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_16.status 2023-12-04T14:28:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_16.status 2023-12-04T14:28:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-12-04T14:28:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:28:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=149 2023-12-04T14:28:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:28:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_16.status 2023-12-04T14:28:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_16.status 2023-12-04T14:28:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-04T14:28:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:37.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 149 = '' 2023-12-04T14:28:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:28:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_16.status 2023-12-04T14:28:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_16.status 2023-12-04T14:28:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-04T14:28:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 149 = '' 2023-12-04T14:28:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:28:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_16.status 2023-12-04T14:28:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_16.status 2023-12-04T14:28:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-04T14:28:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=149 2023-12-04T14:28:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 149 = 149 2023-12-04T14:28:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-12-04T14:28:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:28:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-12-04T14:28:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_16': 2023-12-04T14:28:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:28:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:28:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:28:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3ea48df64 2023-12-04T14:28:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3ea48df64 2023-12-04T14:28:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:28:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:28:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:28:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:28:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:08 2023 2023-12-04T14:28:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:20:08 2023 2023-12-04T14:28:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:08 2023 2023-12-04T14:28:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:28:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:28:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:28:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 5854cc13-6872-45e6-8c82-645c028f7199 2023-12-04T14:28:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:28:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_16 snap 2023-12-04T14:28:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_16 2023-12-04T14:28:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:28:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_16.export 2023-12-04T14:28:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_16.export 2023-12-04T14:28:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_16.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_16.export 2023-12-04T14:28:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_16.export 2023-12-04T14:28:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_16.export 2023-12-04T14:28:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_16.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_16.export 2023-12-04T14:28:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_16.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_16.export 2023-12-04T14:28:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:28:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_17 2023-12-04T14:28:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_17 snap 2023-12-04T14:28:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:28:40.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-12-04T14:28:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-12-04T14:28:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-12-04T14:28:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:28:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-12-04T14:28:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:28:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:28:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:28:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-12-04T14:28:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:28:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:28:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:28:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:28:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:28:42.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:42.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:28:42.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:28:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:28:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:28:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:28:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:28:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:28:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-12-04T14:28:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-12-04T14:28:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:28:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:28:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2023-12-04T14:28:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-12-04T14:28:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2023-12-04T14:28:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:28:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:28:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:28:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:28:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:28:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:28:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2023-12-04T14:28:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-12-04T14:28:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_17 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:28:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:28:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:28:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_17 2023-12-04T14:28:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:28:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:28:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:28:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:28:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_17.status 2023-12-04T14:28:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_17.status 2023-12-04T14:28:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_17.status 2023-12-04T14:28:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_17.status 2023-12-04T14:28:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_17.status 2023-12-04T14:28:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.status 2023-12-04T14:28:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-12-04T14:28:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-12-04T14:28:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 151 2023-12-04T14:28:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_17.status 2023-12-04T14:28:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_17.status 2023-12-04T14:28:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-12-04T14:28:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:28:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=151 2023-12-04T14:28:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:28:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.status 2023-12-04T14:28:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.status 2023-12-04T14:28:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-04T14:28:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=84 2023-12-04T14:28:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 151 = 84 2023-12-04T14:28:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:28:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.status 2023-12-04T14:28:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.status 2023-12-04T14:28:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-04T14:28:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=84 2023-12-04T14:28:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 151 = 84 2023-12-04T14:28:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:28:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.status 2023-12-04T14:28:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.status 2023-12-04T14:28:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-04T14:28:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 151 = '' 2023-12-04T14:28:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:28:45.277 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:28:45.274+0000 7f0527feb640 -1 reset not still connected to 0x5634536561a0 2023-12-04T14:28:45.277 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:28:45.274+0000 7f0527feb640 -1 reset not still connected to 0x563455e995f0 2023-12-04T14:28:45.281 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:28:45.279+0000 7f0527feb640 -1 reset not still connected to 0x563456f651e0 2023-12-04T14:28:45.281 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:28:45.279+0000 7f0527feb640 -1 reset not still connected to 0x56345c40cdd0 2023-12-04T14:28:45.282 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:28:45.279+0000 7f0527feb640 -1 reset not still connected to 0x56345eb28270 2023-12-04T14:28:45.282 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:28:45.279+0000 7f0527feb640 -1 reset not still connected to 0x5634634df6c0 2023-12-04T14:28:45.282 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:28:45.279+0000 7f0527feb640 -1 reset not still connected to 0x563475cbc9c0 2023-12-04T14:28:45.282 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:28:45.279+0000 7f0527feb640 -1 reset not still connected to 0x56347ae2e820 2023-12-04T14:28:45.282 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:28:45.279+0000 7f0527feb640 -1 reset not still connected to 0x56347ec1cea0 2023-12-04T14:28:45.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-12-04T14:28:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.status 2023-12-04T14:28:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.status 2023-12-04T14:28:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-04T14:28:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=151 2023-12-04T14:28:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 151 = 151 2023-12-04T14:28:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-12-04T14:28:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:28:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-12-04T14:28:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_17': 2023-12-04T14:28:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:28:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:28:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:28:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3f05e87aa 2023-12-04T14:28:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3f05e87aa 2023-12-04T14:28:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:28:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:28:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:28:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:28:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:21 2023 2023-12-04T14:28:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:20:21 2023 2023-12-04T14:28:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:21 2023 2023-12-04T14:28:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:28:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:28:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:28:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7bfe12f9-a769-4f59-bca6-8c32a93f3809 2023-12-04T14:28:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:28:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_17 snap 2023-12-04T14:28:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_17 2023-12-04T14:28:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:28:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_17.export 2023-12-04T14:28:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.export 2023-12-04T14:28:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_17.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.export 2023-12-04T14:28:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_17.export 2023-12-04T14:28:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.export 2023-12-04T14:28:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_17.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.export 2023-12-04T14:28:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_17.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_17.export 2023-12-04T14:28:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:28:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_18 2023-12-04T14:28:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_18 snap 2023-12-04T14:28:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:28:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-12-04T14:28:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-12-04T14:28:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:28:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-12-04T14:28:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:28:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:28:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:28:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2023-12-04T14:28:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:28:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:28:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:28:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:28:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:28:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:28:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:28:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:28:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-12-04T14:28:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:28:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:28:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:28:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:28:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:28:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:28:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:28:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:28:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-12-04T14:28:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-04T14:28:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:28:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:28:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:28:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2023-12-04T14:28:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-04T14:28:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2023-12-04T14:28:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:28:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:28:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:28:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:28:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:28:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:28:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2023-12-04T14:28:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-04T14:28:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_18 2023-12-04T14:28:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:28:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:28:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:28:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:28:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:28:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_18 2023-12-04T14:28:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:28:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:28:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:28:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:28:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:28:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:28:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:28:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-12-04T14:28:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:28:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:28:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-12-04T14:28:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_18.status 2023-12-04T14:28:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_18.status 2023-12-04T14:28:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_18.status 2023-12-04T14:28:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_18.status 2023-12-04T14:28:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_18.status 2023-12-04T14:28:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.status 2023-12-04T14:28:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-12-04T14:28:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-12-04T14:28:51.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-12-04T14:28:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 153 2023-12-04T14:28:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_18.status 2023-12-04T14:28:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_18.status 2023-12-04T14:28:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-12-04T14:28:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:28:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=153 2023-12-04T14:28:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:28:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.status 2023-12-04T14:28:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.status 2023-12-04T14:28:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-04T14:28:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=85 2023-12-04T14:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 153 = 85 2023-12-04T14:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:28:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.status 2023-12-04T14:28:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.status 2023-12-04T14:28:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-04T14:28:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=85 2023-12-04T14:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 153 = 85 2023-12-04T14:28:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:28:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.status 2023-12-04T14:28:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.status 2023-12-04T14:28:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-04T14:28:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 153 = '' 2023-12-04T14:28:52.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:52.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:28:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.status 2023-12-04T14:28:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.status 2023-12-04T14:28:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-04T14:28:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=153 2023-12-04T14:28:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 153 = 153 2023-12-04T14:28:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-12-04T14:28:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:28:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-12-04T14:28:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_18': 2023-12-04T14:28:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:28:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:28:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:28:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d391b309d7 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d391b309d7 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:35 2023 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:20:35 2023 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:35 2023 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 9d431cf5-1ffc-4498-b3e0-5bba35366d30 2023-12-04T14:28:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:28:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_18 snap 2023-12-04T14:28:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_18 2023-12-04T14:28:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:28:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_18.export 2023-12-04T14:28:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.export 2023-12-04T14:28:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_18.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.export 2023-12-04T14:28:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_18.export 2023-12-04T14:28:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.export 2023-12-04T14:28:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:28:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_18.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.export 2023-12-04T14:28:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_18.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_18.export 2023-12-04T14:28:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:28:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_19 2023-12-04T14:28:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_19 snap 2023-12-04T14:28:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:28:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:28:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:28:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-12-04T14:28:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:28:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-12-04T14:28:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:28:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:28:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:28:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:28:56.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-12-04T14:28:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-12-04T14:28:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:28:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:28:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:28:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:28:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-12-04T14:28:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:28:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:28:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:28:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:28:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:28:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:28:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:28:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:28:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:28:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-12-04T14:28:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:28:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:28:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:28:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:28:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:28:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:28:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:28:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:28:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:28:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:28:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:28:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-12-04T14:28:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-04T14:28:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2023-12-04T14:28:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:28:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:28:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:28:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:28:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:28:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:28:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:28:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:28:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:28:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:28:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:28:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:28:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:28:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:28:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2023-12-04T14:28:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-04T14:28:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_19 2023-12-04T14:28:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:28:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:28:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:28:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:28:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:28:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:28:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:28:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:28:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:28:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:28:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:28:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:28:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:28:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:28:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:28:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_19 2023-12-04T14:28:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-04T14:28:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:28:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:28:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:28:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_19 2023-12-04T14:28:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:28:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:28:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:28:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:28:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:28:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:28:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:28:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:28:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:28:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:28:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:28:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:28:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:28:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:28:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:28:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:28:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:28:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:28:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:28:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:28:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:28:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:28:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:28:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:28:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:28:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:28:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:28:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:28:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_19 2023-12-04T14:28:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:28:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:28:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:28:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:28:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:28:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:28:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:28:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:28:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-12-04T14:28:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:28:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:28:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:28:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-12-04T14:28:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:28:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:28:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_19.status 2023-12-04T14:28:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_19.status 2023-12-04T14:28:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_19.status 2023-12-04T14:28:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_19.status 2023-12-04T14:28:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_19.status 2023-12-04T14:28:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:28:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.status 2023-12-04T14:28:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-12-04T14:28:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:28:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-12-04T14:28:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 155 2023-12-04T14:28:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_19.status 2023-12-04T14:28:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:28:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:28:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_19.status 2023-12-04T14:28:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-12-04T14:28:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:28:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=155 2023-12-04T14:28:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:28:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.status 2023-12-04T14:28:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:28:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.status 2023-12-04T14:28:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-04T14:28:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=86 2023-12-04T14:28:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 155 = 86 2023-12-04T14:28:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:28:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.status 2023-12-04T14:28:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:28:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.status 2023-12-04T14:28:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-04T14:28:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=86 2023-12-04T14:28:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 155 = 86 2023-12-04T14:28:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:28:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.status 2023-12-04T14:28:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:28:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:28:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:28:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.status 2023-12-04T14:28:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-04T14:28:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:28:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:28:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:28:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:28:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:28:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:28:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:28:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 155 = '' 2023-12-04T14:28:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:28:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:29:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.status 2023-12-04T14:29:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:29:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.status 2023-12-04T14:29:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-04T14:29:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=155 2023-12-04T14:29:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 155 = 155 2023-12-04T14:29:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-12-04T14:29:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:29:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:29:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:01.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-12-04T14:29:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-12-04T14:29:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_19': 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3248df615 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3248df615 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:48 2023 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:20:48 2023 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:48 2023 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:29:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:29:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:29:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 44e764dc-83c2-44b2-bca6-672e14a04ca5 2023-12-04T14:29:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:29:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_19 snap 2023-12-04T14:29:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_19 2023-12-04T14:29:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:29:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_19.export 2023-12-04T14:29:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.export 2023-12-04T14:29:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_19.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.export 2023-12-04T14:29:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_19.export 2023-12-04T14:29:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.export 2023-12-04T14:29:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_19.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.export 2023-12-04T14:29:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_19.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_19.export 2023-12-04T14:29:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:29:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_20 2023-12-04T14:29:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_20 snap 2023-12-04T14:29:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:29:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-12-04T14:29:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-12-04T14:29:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:29:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-12-04T14:29:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:29:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:29:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:29:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-12-04T14:29:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:29:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:29:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:29:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:29:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:29:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:29:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:29:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-04T14:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:29:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-04T14:29:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-04T14:29:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-04T14:29:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:29:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:29:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:29:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:29:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:29:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:29:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-04T14:29:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-04T14:29:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2023-12-04T14:29:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:29:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:29:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:29:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:29:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:29:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:29:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2023-12-04T14:29:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-04T14:29:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:29:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:29:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:29:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2023-12-04T14:29:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:29:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:29:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:29:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:29:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_20.status 2023-12-04T14:29:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_20.status 2023-12-04T14:29:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_20.status 2023-12-04T14:29:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_20.status 2023-12-04T14:29:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_20.status 2023-12-04T14:29:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.status 2023-12-04T14:29:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-12-04T14:29:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:05.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:05.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-12-04T14:29:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 157 2023-12-04T14:29:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_20.status 2023-12-04T14:29:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_20.status 2023-12-04T14:29:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-12-04T14:29:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:29:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=157 2023-12-04T14:29:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:29:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.status 2023-12-04T14:29:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.status 2023-12-04T14:29:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-04T14:29:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=87 2023-12-04T14:29:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 157 = 87 2023-12-04T14:29:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:29:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.status 2023-12-04T14:29:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.status 2023-12-04T14:29:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-04T14:29:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=87 2023-12-04T14:29:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 157 = 87 2023-12-04T14:29:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:29:07.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-12-04T14:29:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.status 2023-12-04T14:29:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.status 2023-12-04T14:29:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-04T14:29:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:29:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:29:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:29:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:29:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 157 = '' 2023-12-04T14:29:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:29:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.status 2023-12-04T14:29:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.status 2023-12-04T14:29:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-04T14:29:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=157 2023-12-04T14:29:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 157 = 157 2023-12-04T14:29:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-12-04T14:29:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:29:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-12-04T14:29:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_20': 2023-12-04T14:29:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:29:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:29:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:29:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3aff73751 2023-12-04T14:29:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3aff73751 2023-12-04T14:29:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:29:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:29:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:29:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:29:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:56 2023 2023-12-04T14:29:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:20:56 2023 2023-12-04T14:29:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:56 2023 2023-12-04T14:29:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:29:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:29:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:29:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: e7fd68b0-bd04-4c3c-af90-fd5517fef987 2023-12-04T14:29:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:29:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_20 snap 2023-12-04T14:29:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_20 2023-12-04T14:29:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:29:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_20.export 2023-12-04T14:29:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.export 2023-12-04T14:29:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_20.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.export 2023-12-04T14:29:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_20.export 2023-12-04T14:29:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.export 2023-12-04T14:29:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_20.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.export 2023-12-04T14:29:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_20.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_20.export 2023-12-04T14:29:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:29:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_21 2023-12-04T14:29:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_21 snap 2023-12-04T14:29:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:29:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-12-04T14:29:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-12-04T14:29:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:29:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-12-04T14:29:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:29:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:29:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:29:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-12-04T14:29:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:29:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:29:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:29:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:29:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:29:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:29:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-12-04T14:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:29:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:29:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:29:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-12-04T14:29:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-12-04T14:29:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:29:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:29:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:29:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2023-12-04T14:29:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-12-04T14:29:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:29:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:29:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2023-12-04T14:29:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-12-04T14:29:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_21 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:29:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:29:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:29:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_21 2023-12-04T14:29:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:29:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:29:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:29:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:29:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_21.status 2023-12-04T14:29:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_21.status 2023-12-04T14:29:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_21.status 2023-12-04T14:29:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_21.status 2023-12-04T14:29:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_21.status 2023-12-04T14:29:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.status 2023-12-04T14:29:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-12-04T14:29:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-12-04T14:29:12.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2023-12-04T14:29:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 159 2023-12-04T14:29:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_21.status 2023-12-04T14:29:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_21.status 2023-12-04T14:29:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-12-04T14:29:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:29:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=159 2023-12-04T14:29:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:29:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.status 2023-12-04T14:29:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.status 2023-12-04T14:29:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-04T14:29:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=88 2023-12-04T14:29:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 159 = 88 2023-12-04T14:29:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:29:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.status 2023-12-04T14:29:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.status 2023-12-04T14:29:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-04T14:29:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=88 2023-12-04T14:29:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 159 = 88 2023-12-04T14:29:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:29:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.status 2023-12-04T14:29:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.status 2023-12-04T14:29:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-04T14:29:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:29:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:29:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:29:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:29:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 159 = '' 2023-12-04T14:29:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:29:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.status 2023-12-04T14:29:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.status 2023-12-04T14:29:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-04T14:29:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=159 2023-12-04T14:29:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 159 = 159 2023-12-04T14:29:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-12-04T14:29:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:29:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-12-04T14:29:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_21': 2023-12-04T14:29:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:29:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:29:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:29:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d32911f4f7 2023-12-04T14:29:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d32911f4f7 2023-12-04T14:29:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:29:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:29:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:29:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:29:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:03 2023 2023-12-04T14:29:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:21:03 2023 2023-12-04T14:29:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:03 2023 2023-12-04T14:29:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:29:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:29:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:29:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 2a201e15-51f0-492a-8b9c-28ef7a490beb 2023-12-04T14:29:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:29:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_21 snap 2023-12-04T14:29:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_21 2023-12-04T14:29:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:29:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_21.export 2023-12-04T14:29:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.export 2023-12-04T14:29:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_21.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.export 2023-12-04T14:29:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_21.export 2023-12-04T14:29:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.export 2023-12-04T14:29:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_21.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.export 2023-12-04T14:29:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_21.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_21.export 2023-12-04T14:29:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:29:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_22 2023-12-04T14:29:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_22 snap 2023-12-04T14:29:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:29:18.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2023-12-04T14:29:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-12-04T14:29:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-12-04T14:29:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:29:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-12-04T14:29:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:29:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:29:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:29:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-12-04T14:29:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:29:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:29:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:29:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:29:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:29:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:29:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:29:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:29:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-12-04T14:29:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:29:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:29:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:29:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:29:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-12-04T14:29:19.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-04T14:29:19.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:19.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:29:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:29:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:29:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2023-12-04T14:29:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-04T14:29:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2023-12-04T14:29:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:29:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:29:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:29:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:29:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:29:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:29:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:29:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:29:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2023-12-04T14:29:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-04T14:29:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_22 2023-12-04T14:29:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:29:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:29:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:29:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:29:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_22 2023-12-04T14:29:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:29:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:29:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:29:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:29:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:29:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:29:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:29:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-12-04T14:29:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:29:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:29:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-12-04T14:29:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_22.status 2023-12-04T14:29:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_22.status 2023-12-04T14:29:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_22.status 2023-12-04T14:29:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_22.status 2023-12-04T14:29:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_22.status 2023-12-04T14:29:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.status 2023-12-04T14:29:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-12-04T14:29:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-12-04T14:29:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 161 2023-12-04T14:29:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_22.status 2023-12-04T14:29:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_22.status 2023-12-04T14:29:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-12-04T14:29:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:29:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=161 2023-12-04T14:29:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:29:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.status 2023-12-04T14:29:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.status 2023-12-04T14:29:20.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-04T14:29:20.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=89 2023-12-04T14:29:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 161 = 89 2023-12-04T14:29:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:29:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.status 2023-12-04T14:29:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.status 2023-12-04T14:29:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-04T14:29:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=89 2023-12-04T14:29:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 161 = 89 2023-12-04T14:29:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:29:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.status 2023-12-04T14:29:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.status 2023-12-04T14:29:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-04T14:29:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:29:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:29:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:29:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:29:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 161 = '' 2023-12-04T14:29:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:29:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.status 2023-12-04T14:29:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.status 2023-12-04T14:29:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-04T14:29:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:23.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-12-04T14:29:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=161 2023-12-04T14:29:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 161 = 161 2023-12-04T14:29:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-12-04T14:29:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:29:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-12-04T14:29:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_22': 2023-12-04T14:29:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:29:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:29:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:29:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3318af3b2 2023-12-04T14:29:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3318af3b2 2023-12-04T14:29:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:29:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:29:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:29:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:29:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:13 2023 2023-12-04T14:29:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:21:13 2023 2023-12-04T14:29:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:13 2023 2023-12-04T14:29:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:29:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:29:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:29:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 059e9df9-d50d-4eb9-8306-2dcc0c369e9c 2023-12-04T14:29:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:29:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_22 snap 2023-12-04T14:29:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_22 2023-12-04T14:29:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:29:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_22.export 2023-12-04T14:29:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.export 2023-12-04T14:29:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_22.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.export 2023-12-04T14:29:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_22.export 2023-12-04T14:29:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.export 2023-12-04T14:29:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_22.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.export 2023-12-04T14:29:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_22.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_22.export 2023-12-04T14:29:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:29:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_23 2023-12-04T14:29:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_23 snap 2023-12-04T14:29:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:29:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-12-04T14:29:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-12-04T14:29:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:29:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-12-04T14:29:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:29:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:29:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:29:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-12-04T14:29:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:29:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:29:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:29:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:29:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:29:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:29:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:29:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:29:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-12-04T14:29:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:29:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:29:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:29:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:29:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:29:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:29:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:29:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:29:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:29:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:29:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:29:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:29:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-12-04T14:29:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-04T14:29:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2023-12-04T14:29:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:29:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:29:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:29:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:29:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:29:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:29:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:29:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:29:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:29:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2023-12-04T14:29:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-04T14:29:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2023-12-04T14:29:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:29:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:29:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:29:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:29:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:29:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:29:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2023-12-04T14:29:27.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-04T14:29:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2023-12-04T14:29:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:29:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:29:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:29:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:29:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:29:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:29:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:29:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:29:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:29:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:29:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:29:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:29:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2023-12-04T14:29:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:29:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:29:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:29:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:29:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-12-04T14:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:29:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:29:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-12-04T14:29:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_23.status 2023-12-04T14:29:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_23.status 2023-12-04T14:29:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_23.status 2023-12-04T14:29:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_23.status 2023-12-04T14:29:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_23.status 2023-12-04T14:29:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_23.status 2023-12-04T14:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-12-04T14:29:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-12-04T14:29:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 163 2023-12-04T14:29:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_23.status 2023-12-04T14:29:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_23.status 2023-12-04T14:29:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-12-04T14:29:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:29:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=163 2023-12-04T14:29:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:29:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_23.status 2023-12-04T14:29:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_23.status 2023-12-04T14:29:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-04T14:29:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:29:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:29:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:29:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:29:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 163 = '' 2023-12-04T14:29:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_23.status 2023-12-04T14:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_23.status 2023-12-04T14:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-04T14:29:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:29:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:29:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:29:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:29:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 163 = '' 2023-12-04T14:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:29:28.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2023-12-04T14:29:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_23.status 2023-12-04T14:29:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_23.status 2023-12-04T14:29:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-04T14:29:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=163 2023-12-04T14:29:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 163 = 163 2023-12-04T14:29:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-12-04T14:29:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:29:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-12-04T14:29:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_23': 2023-12-04T14:29:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:29:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:29:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:29:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d326082b7b 2023-12-04T14:29:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d326082b7b 2023-12-04T14:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:26 2023 2023-12-04T14:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:21:26 2023 2023-12-04T14:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:26 2023 2023-12-04T14:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: e304857c-103b-447a-ae76-3be95494c2ac 2023-12-04T14:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:29:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_23 snap 2023-12-04T14:29:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_23 2023-12-04T14:29:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:29:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_23.export 2023-12-04T14:29:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_23.export 2023-12-04T14:29:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_23.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_23.export 2023-12-04T14:29:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_23.export 2023-12-04T14:29:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_23.export 2023-12-04T14:29:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_23.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_23.export 2023-12-04T14:29:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_23.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_23.export 2023-12-04T14:29:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:29:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_24 2023-12-04T14:29:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_24 snap 2023-12-04T14:29:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:29:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-12-04T14:29:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-12-04T14:29:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:29:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-12-04T14:29:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:29:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:29:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:29:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-12-04T14:29:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:29:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:29:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:29:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:29:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:29:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:29:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:29:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:29:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-04T14:29:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:29:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:29:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:29:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:29:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:29:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:29:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:29:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:29:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:29:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:29:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:29:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:29:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-04T14:29:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-04T14:29:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-04T14:29:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:29:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:29:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:29:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:29:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:29:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-04T14:29:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-04T14:29:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-04T14:29:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:29:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:29:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:29:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:29:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:29:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:29:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:29:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:29:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:29:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:29:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:29:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:29:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-04T14:29:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-04T14:29:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2023-12-04T14:29:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:29:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:29:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:29:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:29:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:29:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:29:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:29:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:29:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:29:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:29:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:29:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:29:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2023-12-04T14:29:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:29:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:29:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:29:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:29:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:29:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:29:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:29:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-12-04T14:29:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:29:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:29:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-12-04T14:29:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_24.status 2023-12-04T14:29:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_24.status 2023-12-04T14:29:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_24.status 2023-12-04T14:29:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_24.status 2023-12-04T14:29:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_24.status 2023-12-04T14:29:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.status 2023-12-04T14:29:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-12-04T14:29:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-12-04T14:29:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 165 2023-12-04T14:29:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_24.status 2023-12-04T14:29:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_24.status 2023-12-04T14:29:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-12-04T14:29:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:29:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=165 2023-12-04T14:29:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:29:34.318 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2023-12-04T14:29:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.status 2023-12-04T14:29:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.status 2023-12-04T14:29:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-04T14:29:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=91 2023-12-04T14:29:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 165 = 91 2023-12-04T14:29:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:29:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.status 2023-12-04T14:29:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.status 2023-12-04T14:29:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-04T14:29:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=91 2023-12-04T14:29:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 165 = 91 2023-12-04T14:29:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:29:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.status 2023-12-04T14:29:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.status 2023-12-04T14:29:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-04T14:29:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:29:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:29:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:29:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:29:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 165 = '' 2023-12-04T14:29:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:29:36.488 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:29:36.486+0000 7f4bfa770640 -1 reset not still connected to 0x56231bf8b1e0 2023-12-04T14:29:36.488 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:29:36.486+0000 7f4bfa770640 -1 reset not still connected to 0x5623261580d0 2023-12-04T14:29:36.499 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:29:36.497+0000 7f4bfa770640 -1 reset not still connected to 0x56233a41e410 2023-12-04T14:29:36.500 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:29:36.497+0000 7f4bfa770640 -1 reset not still connected to 0x56233b9bc5b0 2023-12-04T14:29:36.500 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:29:36.497+0000 7f4bfa770640 -1 reset not still connected to 0x56233d93e820 2023-12-04T14:29:36.500 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:29:36.497+0000 7f4bfa770640 -1 reset not still connected to 0x5623414dc0d0 2023-12-04T14:29:36.502 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:29:36.499+0000 7f4bfa770640 -1 reset not still connected to 0x56234641e750 2023-12-04T14:29:36.502 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:29:36.499+0000 7f4bfa770640 -1 reset not still connected to 0x5623539a7520 2023-12-04T14:29:36.506 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:29:36.504+0000 7f4bfa770640 -1 reset not still connected to 0x562353ea85b0 2023-12-04T14:29:36.507 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:29:36.504+0000 7f4bfa770640 -1 reset not still connected to 0x562353ea8680 2023-12-04T14:29:36.507 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:29:36.504+0000 7f4bfa770640 -1 reset not still connected to 0x562354245d40 2023-12-04T14:29:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.status 2023-12-04T14:29:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.status 2023-12-04T14:29:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-04T14:29:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=165 2023-12-04T14:29:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 165 = 165 2023-12-04T14:29:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-12-04T14:29:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:29:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-12-04T14:29:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_24': 2023-12-04T14:29:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:29:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:29:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:29:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d325210943 2023-12-04T14:29:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d325210943 2023-12-04T14:29:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:29:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:29:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:29:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:29:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:40 2023 2023-12-04T14:29:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:21:40 2023 2023-12-04T14:29:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:40 2023 2023-12-04T14:29:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:29:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:29:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:29:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: eb8df2f9-471f-4269-a2e3-70777410572c 2023-12-04T14:29:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:29:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_24 snap 2023-12-04T14:29:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_24 2023-12-04T14:29:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:29:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_24.export 2023-12-04T14:29:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.export 2023-12-04T14:29:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_24.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.export 2023-12-04T14:29:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_24.export 2023-12-04T14:29:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.export 2023-12-04T14:29:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_24.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.export 2023-12-04T14:29:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_24.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_24.export 2023-12-04T14:29:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:29:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_25 2023-12-04T14:29:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_25 snap 2023-12-04T14:29:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:29:39.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2023-12-04T14:29:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-12-04T14:29:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-12-04T14:29:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:29:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-12-04T14:29:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:29:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:29:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:29:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-12-04T14:29:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:29:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:29:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:29:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:29:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:29:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:29:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:29:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:29:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-12-04T14:29:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:29:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:29:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:29:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:29:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:29:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:29:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-12-04T14:29:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-04T14:29:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2023-12-04T14:29:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:29:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:29:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:29:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:29:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2023-12-04T14:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-04T14:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2023-12-04T14:29:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:29:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:29:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:29:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:29:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:29:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:29:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:29:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:29:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:29:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:29:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:29:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:29:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2023-12-04T14:29:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-04T14:29:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_25 2023-12-04T14:29:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:29:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:29:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:29:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:29:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:29:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:29:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:29:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:29:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_25 2023-12-04T14:29:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:29:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:29:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:29:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:29:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:29:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:29:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:29:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-12-04T14:29:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:29:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:29:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-12-04T14:29:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_25.status 2023-12-04T14:29:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_25.status 2023-12-04T14:29:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_25.status 2023-12-04T14:29:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_25.status 2023-12-04T14:29:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_25.status 2023-12-04T14:29:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_25.status 2023-12-04T14:29:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-12-04T14:29:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-12-04T14:29:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 167 2023-12-04T14:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_25.status 2023-12-04T14:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_25.status 2023-12-04T14:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-12-04T14:29:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:29:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=167 2023-12-04T14:29:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:29:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_25.status 2023-12-04T14:29:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_25.status 2023-12-04T14:29:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-04T14:29:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:42.329 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.28142716248750066) > weight (0.25) 2023-12-04T14:29:42.329 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5419571440248134) > weight (0.25) 2023-12-04T14:29:42.329 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6383947765096532) > weight (0.25) 2023-12-04T14:29:42.329 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-04T14:29:42.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-04T14:29:42.330 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-04T14:29:42.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi176.stderr:2023-12-04T14:29:42.329+0000 7effe6439640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 91139) UID: 1000 2023-12-04T14:29:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 167 = '' 2023-12-04T14:29:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:29:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_25.status 2023-12-04T14:29:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_25.status 2023-12-04T14:29:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-04T14:29:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:29:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:29:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:29:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:29:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 167 = '' 2023-12-04T14:29:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:29:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_25.status 2023-12-04T14:29:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_25.status 2023-12-04T14:29:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-04T14:29:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=167 2023-12-04T14:29:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 167 = 167 2023-12-04T14:29:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-12-04T14:29:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:29:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-12-04T14:29:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_25': 2023-12-04T14:29:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:29:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:29:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:29:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3734b7b9f 2023-12-04T14:29:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3734b7b9f 2023-12-04T14:29:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:29:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:29:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:29:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:29:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:54 2023 2023-12-04T14:29:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:21:54 2023 2023-12-04T14:29:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:54 2023 2023-12-04T14:29:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:29:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:29:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:29:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 44871cb2-b60d-4e0e-8e66-9d9bdcbae254 2023-12-04T14:29:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:29:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_25 snap 2023-12-04T14:29:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_25 2023-12-04T14:29:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:29:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_25.export 2023-12-04T14:29:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_25.export 2023-12-04T14:29:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_25.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_25.export 2023-12-04T14:29:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_25.export 2023-12-04T14:29:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_25.export 2023-12-04T14:29:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_25.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_25.export 2023-12-04T14:29:45.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2023-12-04T14:29:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_25.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_25.export 2023-12-04T14:29:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:29:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_26 2023-12-04T14:29:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_26 snap 2023-12-04T14:29:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:29:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-12-04T14:29:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-12-04T14:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-12-04T14:29:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:29:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:29:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:29:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2023-12-04T14:29:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:29:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:29:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:29:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:29:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:29:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:29:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:29:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:29:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:29:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-12-04T14:29:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-04T14:29:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2023-12-04T14:29:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:29:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:29:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:29:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:29:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:29:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:29:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2023-12-04T14:29:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-04T14:29:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:29:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:29:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:29:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2023-12-04T14:29:47.330 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-04T14:29:47.330 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:29:47.330 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-04T14:29:47.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-04T14:29:47.331 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-04T14:29:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-04T14:29:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2023-12-04T14:29:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:29:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:29:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:29:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:29:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:29:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:29:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:29:47.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-04T14:29:47.336 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2023-12-04T14:29:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2023-12-04T14:29:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:29:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:29:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:29:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:29:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:29:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:29:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_26.status 2023-12-04T14:29:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_26.status 2023-12-04T14:29:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_26.status 2023-12-04T14:29:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_26.status 2023-12-04T14:29:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_26.status 2023-12-04T14:29:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_26.status 2023-12-04T14:29:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-12-04T14:29:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-12-04T14:29:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 169 2023-12-04T14:29:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_26.status 2023-12-04T14:29:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_26.status 2023-12-04T14:29:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-12-04T14:29:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:29:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=169 2023-12-04T14:29:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:29:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_26.status 2023-12-04T14:29:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_26.status 2023-12-04T14:29:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-04T14:29:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:29:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:29:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:29:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:29:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 169 = '' 2023-12-04T14:29:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:29:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_26.status 2023-12-04T14:29:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_26.status 2023-12-04T14:29:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-04T14:29:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:29:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:29:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:29:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:29:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 169 = '' 2023-12-04T14:29:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:29:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_26.status 2023-12-04T14:29:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_26.status 2023-12-04T14:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-04T14:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=169 2023-12-04T14:29:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 169 = 169 2023-12-04T14:29:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-12-04T14:29:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:29:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:50.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2023-12-04T14:29:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_26': 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d389de6656 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d389de6656 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:07 2023 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:07 2023 2023-12-04T14:29:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:07 2023 2023-12-04T14:29:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:29:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:29:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:29:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: edd4af1f-3c2b-4169-adc5-5d59c2cc5a61 2023-12-04T14:29:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:29:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_26 snap 2023-12-04T14:29:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_26 2023-12-04T14:29:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:29:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_26.export 2023-12-04T14:29:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_26.export 2023-12-04T14:29:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_26.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_26.export 2023-12-04T14:29:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_26.export 2023-12-04T14:29:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_26.export 2023-12-04T14:29:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_26.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_26.export 2023-12-04T14:29:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_26.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_26.export 2023-12-04T14:29:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:29:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_27 2023-12-04T14:29:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_27 snap 2023-12-04T14:29:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:29:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-12-04T14:29:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-12-04T14:29:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:29:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-12-04T14:29:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:29:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:29:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:29:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2023-12-04T14:29:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:29:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:29:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:29:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:29:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:29:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:29:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:29:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:29:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-12-04T14:29:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:29:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:29:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:29:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:29:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:29:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:29:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-12-04T14:29:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-12-04T14:29:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-12-04T14:29:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:29:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:29:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:29:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:29:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:29:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:29:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-12-04T14:29:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-12-04T14:29:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2023-12-04T14:29:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:29:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:29:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:29:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:29:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:29:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:29:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2023-12-04T14:29:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-12-04T14:29:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:29:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:29:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:29:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2023-12-04T14:29:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:29:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:29:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:29:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:29:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_27.status 2023-12-04T14:29:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_27.status 2023-12-04T14:29:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_27.status 2023-12-04T14:29:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_27.status 2023-12-04T14:29:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_27.status 2023-12-04T14:29:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_27.status 2023-12-04T14:29:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-12-04T14:29:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-12-04T14:29:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 171 2023-12-04T14:29:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_27.status 2023-12-04T14:29:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_27.status 2023-12-04T14:29:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-12-04T14:29:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:29:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=171 2023-12-04T14:29:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:29:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_27.status 2023-12-04T14:29:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_27.status 2023-12-04T14:29:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-04T14:29:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:29:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:29:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:29:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:29:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 171 = '' 2023-12-04T14:29:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:29:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_27.status 2023-12-04T14:29:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_27.status 2023-12-04T14:29:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-04T14:29:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:29:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:29:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:29:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:29:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:29:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 171 = '' 2023-12-04T14:29:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:29:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:29:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_27.status 2023-12-04T14:29:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_27.status 2023-12-04T14:29:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-04T14:29:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:29:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:29:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=171 2023-12-04T14:29:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 171 = 171 2023-12-04T14:29:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-12-04T14:29:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:29:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:55.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2023-12-04T14:29:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-12-04T14:29:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_27': 2023-12-04T14:29:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:29:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:29:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:29:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3fe9fbf08 2023-12-04T14:29:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3fe9fbf08 2023-12-04T14:29:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:29:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:29:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:29:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:29:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:13 2023 2023-12-04T14:29:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:13 2023 2023-12-04T14:29:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:13 2023 2023-12-04T14:29:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:29:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:29:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:29:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: e393e7c2-3dbe-4e5b-9517-da1c79d5fe47 2023-12-04T14:29:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:29:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_27 snap 2023-12-04T14:29:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_27 2023-12-04T14:29:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:29:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_27.export 2023-12-04T14:29:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_27.export 2023-12-04T14:29:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_27.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_27.export 2023-12-04T14:29:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_27.export 2023-12-04T14:29:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_27.export 2023-12-04T14:29:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:29:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_27.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_27.export 2023-12-04T14:29:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_27.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_27.export 2023-12-04T14:29:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:29:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_28 2023-12-04T14:29:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_28 snap 2023-12-04T14:29:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:29:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:29:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:29:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-12-04T14:29:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:29:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-12-04T14:29:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:29:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:29:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:29:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:29:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-12-04T14:29:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:29:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:29:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:29:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:29:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:29:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2023-12-04T14:29:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:29:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:29:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:29:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:29:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:29:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:29:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:29:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:29:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:29:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-12-04T14:29:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:29:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:29:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:29:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:29:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:29:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:29:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:29:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:29:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:29:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:29:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-12-04T14:29:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-04T14:29:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2023-12-04T14:29:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:29:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:29:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:29:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:29:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:29:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:29:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:29:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:29:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:29:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2023-12-04T14:29:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-04T14:29:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2023-12-04T14:29:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:29:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:29:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:29:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:29:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:29:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:29:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:29:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:29:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:29:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2023-12-04T14:29:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-04T14:29:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:29:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:29:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_28 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:29:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:29:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:29:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:29:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_28 2023-12-04T14:29:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:29:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:29:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:29:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:29:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:29:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:29:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:29:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:29:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-12-04T14:29:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:29:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:29:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:29:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-12-04T14:29:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:29:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:29:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_28.status 2023-12-04T14:29:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_28.status 2023-12-04T14:29:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_28.status 2023-12-04T14:29:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_28.status 2023-12-04T14:29:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_28.status 2023-12-04T14:29:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:29:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_28.status 2023-12-04T14:29:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-12-04T14:29:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:29:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:29:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:29:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-12-04T14:30:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 173 2023-12-04T14:30:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_28.status 2023-12-04T14:30:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:30:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_28.status 2023-12-04T14:30:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-12-04T14:30:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:30:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=173 2023-12-04T14:30:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:30:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_28.status 2023-12-04T14:30:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:30:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_28.status 2023-12-04T14:30:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-04T14:30:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 173 = '' 2023-12-04T14:30:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:30:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_28.status 2023-12-04T14:30:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:30:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_28.status 2023-12-04T14:30:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-04T14:30:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:00.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 173 = '' 2023-12-04T14:30:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:30:01.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2023-12-04T14:30:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_28.status 2023-12-04T14:30:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:30:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_28.status 2023-12-04T14:30:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-04T14:30:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=173 2023-12-04T14:30:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 173 = 173 2023-12-04T14:30:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-12-04T14:30:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:30:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:30:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-12-04T14:30:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_28': 2023-12-04T14:30:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:30:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:30:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:30:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3a7b7e84b 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3a7b7e84b 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:25 2023 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:25 2023 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:25 2023 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: ad2b0660-9c28-48f0-8b26-08f4860ad09e 2023-12-04T14:30:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:30:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_28 snap 2023-12-04T14:30:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_28 2023-12-04T14:30:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:30:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_28.export 2023-12-04T14:30:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_28.export 2023-12-04T14:30:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_28.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_28.export 2023-12-04T14:30:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_28.export 2023-12-04T14:30:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_28.export 2023-12-04T14:30:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_28.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_28.export 2023-12-04T14:30:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_28.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_28.export 2023-12-04T14:30:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:30:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_29 2023-12-04T14:30:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_29 snap 2023-12-04T14:30:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:30:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-12-04T14:30:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-12-04T14:30:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:30:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-12-04T14:30:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:30:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:30:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:30:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-12-04T14:30:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:30:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:30:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:30:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:30:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:30:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:30:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:30:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:30:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-12-04T14:30:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:30:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:30:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:30:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:30:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:30:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:30:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-12-04T14:30:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-04T14:30:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-12-04T14:30:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:30:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:30:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:30:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-12-04T14:30:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-04T14:30:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:30:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:30:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:30:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2023-12-04T14:30:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-04T14:30:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2023-12-04T14:30:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:30:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:30:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:30:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:30:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:30:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:30:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2023-12-04T14:30:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:30:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:30:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:30:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:30:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:30:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:30:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_29.status 2023-12-04T14:30:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_29.status 2023-12-04T14:30:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_29.status 2023-12-04T14:30:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_29.status 2023-12-04T14:30:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_29.status 2023-12-04T14:30:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_29.status 2023-12-04T14:30:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-12-04T14:30:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-12-04T14:30:05.750 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-04T14:30:05.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-04T14:30:05.750 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-04T14:30:05.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi176.stderr:2023-12-04T14:30:05.749+0000 7ff1f4144640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 104511) UID: 1000 2023-12-04T14:30:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 175 2023-12-04T14:30:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_29.status 2023-12-04T14:30:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_29.status 2023-12-04T14:30:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-12-04T14:30:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:30:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=175 2023-12-04T14:30:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:30:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_29.status 2023-12-04T14:30:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_29.status 2023-12-04T14:30:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-04T14:30:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 175 = '' 2023-12-04T14:30:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:30:06.741 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2023-12-04T14:30:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_29.status 2023-12-04T14:30:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_29.status 2023-12-04T14:30:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-04T14:30:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 175 = '' 2023-12-04T14:30:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:30:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_29.status 2023-12-04T14:30:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_29.status 2023-12-04T14:30:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-04T14:30:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=175 2023-12-04T14:30:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 175 = 175 2023-12-04T14:30:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-12-04T14:30:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:30:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-12-04T14:30:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_29': 2023-12-04T14:30:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:30:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:30:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:30:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d39517f19e 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d39517f19e 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:39 2023 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:39 2023 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:39 2023 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 040b1033-d414-4ab1-bb84-83c330f7ff65 2023-12-04T14:30:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:30:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_29 snap 2023-12-04T14:30:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_29 2023-12-04T14:30:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:30:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_29.export 2023-12-04T14:30:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_29.export 2023-12-04T14:30:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_29.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_29.export 2023-12-04T14:30:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_29.export 2023-12-04T14:30:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:09.097 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_29.export 2023-12-04T14:30:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_29.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_29.export 2023-12-04T14:30:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_29.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_29.export 2023-12-04T14:30:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_30 2023-12-04T14:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_30 snap 2023-12-04T14:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:30:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-12-04T14:30:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-12-04T14:30:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:30:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-12-04T14:30:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:30:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:30:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:30:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2023-12-04T14:30:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:30:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:30:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:30:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:30:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:30:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:30:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:30:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:30:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-12-04T14:30:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:30:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:30:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:30:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:30:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:30:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:30:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:30:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:30:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:30:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:30:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:30:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:30:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-12-04T14:30:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:30:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:30:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:30:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-12-04T14:30:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-04T14:30:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2023-12-04T14:30:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:30:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:30:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:30:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:30:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:30:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:30:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2023-12-04T14:30:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-04T14:30:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2023-12-04T14:30:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:30:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:30:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:30:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:30:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:30:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:30:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2023-12-04T14:30:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:30:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:30:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:30:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:30:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:30:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:30:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:30:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-12-04T14:30:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:11.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:30:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:30:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-12-04T14:30:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_30.status 2023-12-04T14:30:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_30.status 2023-12-04T14:30:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_30.status 2023-12-04T14:30:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_30.status 2023-12-04T14:30:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_30.status 2023-12-04T14:30:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.status 2023-12-04T14:30:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-12-04T14:30:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-12-04T14:30:12.144 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2023-12-04T14:30:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 177 2023-12-04T14:30:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_30.status 2023-12-04T14:30:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_30.status 2023-12-04T14:30:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-12-04T14:30:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:30:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=177 2023-12-04T14:30:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:30:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.status 2023-12-04T14:30:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.status 2023-12-04T14:30:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-04T14:30:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=97 2023-12-04T14:30:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 177 = 97 2023-12-04T14:30:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:30:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.status 2023-12-04T14:30:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.status 2023-12-04T14:30:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-04T14:30:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 177 = '' 2023-12-04T14:30:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:30:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.status 2023-12-04T14:30:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.status 2023-12-04T14:30:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-04T14:30:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 177 = '' 2023-12-04T14:30:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:30:14.751 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-04T14:30:14.751 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:30:14.751 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-04T14:30:14.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-04T14:30:14.751 DEBUG:teuthology.orchestra.run.smithi176:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-04T14:30:14.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-04T14:30:14.754 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2023-12-04T14:30:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.status 2023-12-04T14:30:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.status 2023-12-04T14:30:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-04T14:30:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=177 2023-12-04T14:30:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 177 = 177 2023-12-04T14:30:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-12-04T14:30:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:30:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-12-04T14:30:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_30': 2023-12-04T14:30:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:30:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:30:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:30:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3ffbd15f1 2023-12-04T14:30:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3ffbd15f1 2023-12-04T14:30:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:30:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:30:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:30:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:30:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:46 2023 2023-12-04T14:30:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:46 2023 2023-12-04T14:30:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:46 2023 2023-12-04T14:30:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:30:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:30:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:30:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 233226c3-03eb-48bb-bc7e-fe8f5e6be5a9 2023-12-04T14:30:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:30:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_30 snap 2023-12-04T14:30:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_30 2023-12-04T14:30:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:30:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_30.export 2023-12-04T14:30:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.export 2023-12-04T14:30:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_30.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.export 2023-12-04T14:30:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_30.export 2023-12-04T14:30:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.export 2023-12-04T14:30:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_30.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.export 2023-12-04T14:30:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_30.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_30.export 2023-12-04T14:30:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:30:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_31 2023-12-04T14:30:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_31 snap 2023-12-04T14:30:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:30:17.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2023-12-04T14:30:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-12-04T14:30:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-12-04T14:30:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:30:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-12-04T14:30:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:30:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:30:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:30:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-12-04T14:30:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:30:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:30:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:30:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:30:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:30:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:30:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:30:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:30:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-12-04T14:30:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:30:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:30:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:30:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:30:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:30:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:30:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:30:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:30:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-12-04T14:30:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-04T14:30:19.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:19.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2023-12-04T14:30:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:30:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:30:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:30:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:30:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:30:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:30:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2023-12-04T14:30:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-04T14:30:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:30:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:30:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:30:19.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2023-12-04T14:30:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-04T14:30:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:19.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_31 2023-12-04T14:30:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:30:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:30:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:30:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:30:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:30:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:30:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_31 2023-12-04T14:30:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:30:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:30:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:30:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:30:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:30:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:30:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:30:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-12-04T14:30:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:30:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:30:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-12-04T14:30:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_31.status 2023-12-04T14:30:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_31.status 2023-12-04T14:30:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_31.status 2023-12-04T14:30:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_31.status 2023-12-04T14:30:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_31.status 2023-12-04T14:30:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.status 2023-12-04T14:30:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-12-04T14:30:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-12-04T14:30:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 179 2023-12-04T14:30:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_31.status 2023-12-04T14:30:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_31.status 2023-12-04T14:30:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-12-04T14:30:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:30:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=179 2023-12-04T14:30:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:30:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.status 2023-12-04T14:30:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.status 2023-12-04T14:30:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-04T14:30:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=98 2023-12-04T14:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 179 = 98 2023-12-04T14:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:30:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.status 2023-12-04T14:30:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.status 2023-12-04T14:30:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-04T14:30:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=98 2023-12-04T14:30:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 179 = 98 2023-12-04T14:30:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:30:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.status 2023-12-04T14:30:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.status 2023-12-04T14:30:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-04T14:30:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 179 = '' 2023-12-04T14:30:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:30:22.952 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2023-12-04T14:30:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.status 2023-12-04T14:30:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.status 2023-12-04T14:30:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-04T14:30:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=179 2023-12-04T14:30:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 179 = 179 2023-12-04T14:30:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-12-04T14:30:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:30:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-12-04T14:30:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_31': 2023-12-04T14:30:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:30:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:30:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:30:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3518272ec 2023-12-04T14:30:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3518272ec 2023-12-04T14:30:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:30:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:30:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:30:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:30:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:55 2023 2023-12-04T14:30:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:22:55 2023 2023-12-04T14:30:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:55 2023 2023-12-04T14:30:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:30:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:30:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:30:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 55aa29aa-3f60-443f-a765-b4d1a2257a41 2023-12-04T14:30:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:30:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_31 snap 2023-12-04T14:30:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_31 2023-12-04T14:30:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:30:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_31.export 2023-12-04T14:30:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.export 2023-12-04T14:30:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_31.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.export 2023-12-04T14:30:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_31.export 2023-12-04T14:30:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.export 2023-12-04T14:30:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_31.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.export 2023-12-04T14:30:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_31.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_31.export 2023-12-04T14:30:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:30:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_32 2023-12-04T14:30:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_32 snap 2023-12-04T14:30:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:30:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-12-04T14:30:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-12-04T14:30:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:30:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-12-04T14:30:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:30:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:30:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:30:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2023-12-04T14:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:30:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-12-04T14:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:30:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:30:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:30:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-12-04T14:30:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-12-04T14:30:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:30:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:30:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:30:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-12-04T14:30:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-12-04T14:30:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2023-12-04T14:30:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:30:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:30:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:30:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:30:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:30:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:30:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:30:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2023-12-04T14:30:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-12-04T14:30:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2023-12-04T14:30:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:30:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:30:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:30:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:30:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2023-12-04T14:30:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:30:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:30:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:30:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:30:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:30:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:30:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:30:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-12-04T14:30:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:30:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:30:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-12-04T14:30:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_32.status 2023-12-04T14:30:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_32.status 2023-12-04T14:30:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_32.status 2023-12-04T14:30:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_32.status 2023-12-04T14:30:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_32.status 2023-12-04T14:30:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_32.status 2023-12-04T14:30:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-12-04T14:30:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-12-04T14:30:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 181 2023-12-04T14:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_32.status 2023-12-04T14:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_32.status 2023-12-04T14:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-12-04T14:30:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:30:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=181 2023-12-04T14:30:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:30:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_32.status 2023-12-04T14:30:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_32.status 2023-12-04T14:30:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-04T14:30:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:28.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2023-12-04T14:30:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 181 = '' 2023-12-04T14:30:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:30:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_32.status 2023-12-04T14:30:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_32.status 2023-12-04T14:30:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-04T14:30:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 181 = '' 2023-12-04T14:30:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:30:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_32.status 2023-12-04T14:30:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_32.status 2023-12-04T14:30:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-04T14:30:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=181 2023-12-04T14:30:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 181 = 181 2023-12-04T14:30:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-12-04T14:30:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:30:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-12-04T14:30:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_32': 2023-12-04T14:30:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:30:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:30:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:30:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d36e345418 2023-12-04T14:30:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d36e345418 2023-12-04T14:30:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:30:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:30:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:30:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:30:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:01 2023 2023-12-04T14:30:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:23:01 2023 2023-12-04T14:30:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:01 2023 2023-12-04T14:30:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:30:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:30:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:30:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: da0438b0-0666-4c5d-bca2-497b527c3c8a 2023-12-04T14:30:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:30:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_32 snap 2023-12-04T14:30:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_32 2023-12-04T14:30:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:30:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_32.export 2023-12-04T14:30:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_32.export 2023-12-04T14:30:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_32.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_32.export 2023-12-04T14:30:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_32.export 2023-12-04T14:30:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_32.export 2023-12-04T14:30:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_32.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_32.export 2023-12-04T14:30:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_32.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_32.export 2023-12-04T14:30:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:30:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_33 2023-12-04T14:30:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_33 snap 2023-12-04T14:30:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:30:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-12-04T14:30:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-12-04T14:30:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:30:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-12-04T14:30:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:30:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:30:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:30:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2023-12-04T14:30:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:30:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:30:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:30:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:30:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:30:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:30:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:30:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:30:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-12-04T14:30:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:30:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:30:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:30:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:30:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:30:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:30:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:30:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:30:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:30:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:30:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:30:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:30:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-12-04T14:30:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-04T14:30:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-12-04T14:30:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:30:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:30:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:30:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:30:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:30:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:30:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:30:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:30:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:30:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:30:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:30:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:30:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-12-04T14:30:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-04T14:30:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2023-12-04T14:30:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:30:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:30:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:30:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:30:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:30:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:30:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:30:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:30:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:30:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:30:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:30:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:30:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2023-12-04T14:30:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-04T14:30:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2023-12-04T14:30:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:30:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:30:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:30:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:30:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:30:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:30:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:30:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:30:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:30:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:30:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:30:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:30:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2023-12-04T14:30:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:30:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:30:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:30:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:30:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:30:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:30:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:30:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-12-04T14:30:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:30:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:30:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-12-04T14:30:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_33.status 2023-12-04T14:30:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_33.status 2023-12-04T14:30:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_33.status 2023-12-04T14:30:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_33.status 2023-12-04T14:30:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_33.status 2023-12-04T14:30:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.status 2023-12-04T14:30:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-12-04T14:30:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-12-04T14:30:33.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2023-12-04T14:30:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 183 2023-12-04T14:30:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_33.status 2023-12-04T14:30:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_33.status 2023-12-04T14:30:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-12-04T14:30:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=183 2023-12-04T14:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:30:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.status 2023-12-04T14:30:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.status 2023-12-04T14:30:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-04T14:30:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=100 2023-12-04T14:30:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 183 = 100 2023-12-04T14:30:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:30:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.status 2023-12-04T14:30:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.status 2023-12-04T14:30:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-04T14:30:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=100 2023-12-04T14:30:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 183 = 100 2023-12-04T14:30:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:30:35.495 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:35.493+0000 7f0527feb640 -1 reset not still connected to 0x563435bb5520 2023-12-04T14:30:35.495 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:35.493+0000 7f0527feb640 -1 reset not still connected to 0x56343644d450 2023-12-04T14:30:35.495 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:35.493+0000 7f0527feb640 -1 reset not still connected to 0x563442b30820 2023-12-04T14:30:35.495 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:35.493+0000 7f0527feb640 -1 reset not still connected to 0x5634715900d0 2023-12-04T14:30:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.status 2023-12-04T14:30:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.status 2023-12-04T14:30:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-04T14:30:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 183 = '' 2023-12-04T14:30:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1.6 2023-12-04T14:30:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.status 2023-12-04T14:30:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.status 2023-12-04T14:30:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-04T14:30:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=183 2023-12-04T14:30:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 183 = 183 2023-12-04T14:30:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-12-04T14:30:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:30:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-12-04T14:30:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_33': 2023-12-04T14:30:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:30:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:30:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:30:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3bd65a8c0 2023-12-04T14:30:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3bd65a8c0 2023-12-04T14:30:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:30:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:30:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:30:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:30:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:11 2023 2023-12-04T14:30:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:23:11 2023 2023-12-04T14:30:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:11 2023 2023-12-04T14:30:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:30:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:30:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:30:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 709187ee-c3f5-47e9-a97c-a08560690fe3 2023-12-04T14:30:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:30:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_33 snap 2023-12-04T14:30:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_33 2023-12-04T14:30:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:30:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_33.export 2023-12-04T14:30:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.export 2023-12-04T14:30:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_33.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.export 2023-12-04T14:30:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_33.export 2023-12-04T14:30:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.export 2023-12-04T14:30:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_33.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.export 2023-12-04T14:30:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_33.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_33.export 2023-12-04T14:30:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:30:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_34 2023-12-04T14:30:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_34 snap 2023-12-04T14:30:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:30:39.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2023-12-04T14:30:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-12-04T14:30:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-12-04T14:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-12-04T14:30:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:30:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:30:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:30:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-12-04T14:30:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:30:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:30:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:30:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:30:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:30:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:30:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:30:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:30:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-04T14:30:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:30:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:30:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:30:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:30:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:30:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:30:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:30:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:30:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:30:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:30:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:30:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:30:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-04T14:30:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-04T14:30:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-04T14:30:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:30:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:30:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:30:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:30:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-04T14:30:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-04T14:30:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2023-12-04T14:30:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:30:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:30:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:30:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:30:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:30:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:30:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:30:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:30:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:30:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:30:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:30:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:30:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2023-12-04T14:30:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-04T14:30:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2023-12-04T14:30:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:30:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:30:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:30:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:30:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:30:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:30:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:30:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:30:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:30:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:30:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:30:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:30:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2023-12-04T14:30:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:30:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:30:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:30:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:30:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:30:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:30:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:30:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-12-04T14:30:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:30:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:30:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-12-04T14:30:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_34.status 2023-12-04T14:30:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_34.status 2023-12-04T14:30:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_34.status 2023-12-04T14:30:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_34.status 2023-12-04T14:30:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_34.status 2023-12-04T14:30:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_34.status 2023-12-04T14:30:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-12-04T14:30:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-12-04T14:30:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 185 2023-12-04T14:30:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_34.status 2023-12-04T14:30:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_34.status 2023-12-04T14:30:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-12-04T14:30:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=185 2023-12-04T14:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:30:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_34.status 2023-12-04T14:30:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_34.status 2023-12-04T14:30:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-04T14:30:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 185 = '' 2023-12-04T14:30:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:30:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_34.status 2023-12-04T14:30:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_34.status 2023-12-04T14:30:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-04T14:30:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 185 = '' 2023-12-04T14:30:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:30:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_34.status 2023-12-04T14:30:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_34.status 2023-12-04T14:30:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-04T14:30:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=185 2023-12-04T14:30:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 185 = 185 2023-12-04T14:30:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-12-04T14:30:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:30:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:44.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2023-12-04T14:30:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-12-04T14:30:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_34': 2023-12-04T14:30:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:30:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:30:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:30:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d32bdab1ea 2023-12-04T14:30:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d32bdab1ea 2023-12-04T14:30:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:30:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:30:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:30:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:30:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:25 2023 2023-12-04T14:30:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:23:25 2023 2023-12-04T14:30:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:25 2023 2023-12-04T14:30:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:30:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:30:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:30:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 53abed9b-8d5d-4a15-a122-712b8dc9c3d5 2023-12-04T14:30:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:30:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_34 snap 2023-12-04T14:30:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_34 2023-12-04T14:30:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:30:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_34.export 2023-12-04T14:30:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_34.export 2023-12-04T14:30:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_34.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_34.export 2023-12-04T14:30:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_34.export 2023-12-04T14:30:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_34.export 2023-12-04T14:30:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_34.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_34.export 2023-12-04T14:30:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_34.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_34.export 2023-12-04T14:30:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:30:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_35 2023-12-04T14:30:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_35 snap 2023-12-04T14:30:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:30:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-12-04T14:30:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-12-04T14:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-12-04T14:30:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:30:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:30:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:30:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2023-12-04T14:30:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:30:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:30:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:30:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:30:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:30:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:30:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:30:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:30:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-12-04T14:30:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:30:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:30:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:30:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:30:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:30:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:30:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:30:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:30:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:30:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:30:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:30:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:30:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-12-04T14:30:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-12-04T14:30:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2023-12-04T14:30:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:30:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:30:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:30:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:30:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:30:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:30:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:30:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:30:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:30:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:30:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:30:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:30:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2023-12-04T14:30:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-12-04T14:30:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2023-12-04T14:30:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:30:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:30:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:30:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:30:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:30:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:30:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:30:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:30:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:30:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:30:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:30:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:30:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2023-12-04T14:30:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-12-04T14:30:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_35 2023-12-04T14:30:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:30:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:30:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:30:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:30:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:30:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:30:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:30:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:30:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:30:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:30:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:30:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:30:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_35 2023-12-04T14:30:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:30:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:30:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:30:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:30:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:30:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:30:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:30:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-12-04T14:30:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:30:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:30:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-12-04T14:30:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_35.status 2023-12-04T14:30:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_35.status 2023-12-04T14:30:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_35.status 2023-12-04T14:30:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_35.status 2023-12-04T14:30:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_35.status 2023-12-04T14:30:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_35.status 2023-12-04T14:30:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-12-04T14:30:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-12-04T14:30:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 187 2023-12-04T14:30:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_35.status 2023-12-04T14:30:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_35.status 2023-12-04T14:30:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-12-04T14:30:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:30:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=187 2023-12-04T14:30:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:30:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_35.status 2023-12-04T14:30:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_35.status 2023-12-04T14:30:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-04T14:30:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 187 = '' 2023-12-04T14:30:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:30:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_35.status 2023-12-04T14:30:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_35.status 2023-12-04T14:30:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-04T14:30:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 187 = '' 2023-12-04T14:30:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:30:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_35.status 2023-12-04T14:30:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_35.status 2023-12-04T14:30:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-04T14:30:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=187 2023-12-04T14:30:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 187 = 187 2023-12-04T14:30:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-12-04T14:30:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:30:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:49.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2023-12-04T14:30:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-12-04T14:30:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_35': 2023-12-04T14:30:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:30:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:30:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:30:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d38f87952b 2023-12-04T14:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d38f87952b 2023-12-04T14:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:37 2023 2023-12-04T14:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:23:37 2023 2023-12-04T14:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:37 2023 2023-12-04T14:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:30:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:30:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:30:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 75dc1148-0c6b-4bf4-962c-9f4dc15daa97 2023-12-04T14:30:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:30:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_35 snap 2023-12-04T14:30:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_35 2023-12-04T14:30:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:30:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_35.export 2023-12-04T14:30:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_35.export 2023-12-04T14:30:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_35.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_35.export 2023-12-04T14:30:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_35.export 2023-12-04T14:30:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_35.export 2023-12-04T14:30:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_35.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_35.export 2023-12-04T14:30:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_35.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_35.export 2023-12-04T14:30:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:30:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_36 2023-12-04T14:30:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_36 snap 2023-12-04T14:30:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:30:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-12-04T14:30:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-12-04T14:30:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:30:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-12-04T14:30:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:30:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:30:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:30:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2023-12-04T14:30:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:30:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:30:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:30:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:30:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:30:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:30:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:30:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:30:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-12-04T14:30:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:30:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:30:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:30:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:30:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:30:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:30:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-12-04T14:30:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-04T14:30:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:30:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:30:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:30:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2023-12-04T14:30:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-04T14:30:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:30:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:30:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:30:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:30:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2023-12-04T14:30:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-04T14:30:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2023-12-04T14:30:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:30:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:30:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:30:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:30:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:30:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:30:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2023-12-04T14:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:30:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-12-04T14:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:30:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-12-04T14:30:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_36.status 2023-12-04T14:30:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_36.status 2023-12-04T14:30:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_36.status 2023-12-04T14:30:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_36.status 2023-12-04T14:30:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_36.status 2023-12-04T14:30:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_36.status 2023-12-04T14:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-12-04T14:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-12-04T14:30:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 189 2023-12-04T14:30:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_36.status 2023-12-04T14:30:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_36.status 2023-12-04T14:30:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-12-04T14:30:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:30:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=189 2023-12-04T14:30:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:30:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_36.status 2023-12-04T14:30:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_36.status 2023-12-04T14:30:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-04T14:30:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 189 = '' 2023-12-04T14:30:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:30:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_36.status 2023-12-04T14:30:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_36.status 2023-12-04T14:30:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-04T14:30:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:30:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:30:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:30:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:30:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:30:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 189 = '' 2023-12-04T14:30:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:30:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:30:55.381 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2023-12-04T14:30:55.479 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.476+0000 7f0527feb640 -1 reset not still connected to 0x56343f9a9930 2023-12-04T14:30:55.484 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.483+0000 7f0527feb640 -1 reset not still connected to 0x56343fa940d0 2023-12-04T14:30:55.488 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.487+0000 7f0527feb640 -1 reset not still connected to 0x563441f55ad0 2023-12-04T14:30:55.488 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.487+0000 7f0527feb640 -1 reset not still connected to 0x56344800b380 2023-12-04T14:30:55.492 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.491+0000 7f0527feb640 -1 reset not still connected to 0x56344946d520 2023-12-04T14:30:55.492 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.491+0000 7f0527feb640 -1 reset not still connected to 0x563449c52340 2023-12-04T14:30:55.492 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.491+0000 7f0527feb640 -1 reset not still connected to 0x56344b43da00 2023-12-04T14:30:55.492 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.491+0000 7f0527feb640 -1 reset not still connected to 0x56344b8336c0 2023-12-04T14:30:55.492 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.491+0000 7f0527feb640 -1 reset not still connected to 0x563451553930 2023-12-04T14:30:55.493 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.491+0000 7f0527feb640 -1 reset not still connected to 0x563451e171e0 2023-12-04T14:30:55.493 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.491+0000 7f0527feb640 -1 reset not still connected to 0x5634536561a0 2023-12-04T14:30:55.493 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.491+0000 7f0527feb640 -1 reset not still connected to 0x563455e995f0 2023-12-04T14:30:55.493 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.491+0000 7f0527feb640 -1 reset not still connected to 0x563456f651e0 2023-12-04T14:30:55.493 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.491+0000 7f0527feb640 -1 reset not still connected to 0x56345c40cdd0 2023-12-04T14:30:55.493 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.491+0000 7f0527feb640 -1 reset not still connected to 0x56345eb28270 2023-12-04T14:30:55.493 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.491+0000 7f0527feb640 -1 reset not still connected to 0x5634634df6c0 2023-12-04T14:30:55.493 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.493+0000 7f0527feb640 -1 reset not still connected to 0x563475cbc9c0 2023-12-04T14:30:55.494 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.493+0000 7f0527feb640 -1 reset not still connected to 0x56347ae2e820 2023-12-04T14:30:55.494 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:30:55.493+0000 7f0527feb640 -1 reset not still connected to 0x56347ec1cea0 2023-12-04T14:30:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_36.status 2023-12-04T14:30:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_36.status 2023-12-04T14:30:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-04T14:30:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:30:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:30:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=189 2023-12-04T14:30:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 189 = 189 2023-12-04T14:30:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-12-04T14:30:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:30:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-12-04T14:30:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_36': 2023-12-04T14:30:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:30:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:30:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:30:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3b8f9e819 2023-12-04T14:30:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3b8f9e819 2023-12-04T14:30:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:30:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:30:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:30:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:30:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:45 2023 2023-12-04T14:30:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:23:45 2023 2023-12-04T14:30:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:45 2023 2023-12-04T14:30:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:30:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:30:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:30:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: c88f1ade-8455-43f9-be6e-45b69b6f15a1 2023-12-04T14:30:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:30:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_36 snap 2023-12-04T14:30:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_36 2023-12-04T14:30:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:30:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_36.export 2023-12-04T14:30:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_36.export 2023-12-04T14:30:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_36.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_36.export 2023-12-04T14:30:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_36.export 2023-12-04T14:30:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_36.export 2023-12-04T14:30:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-04T14:30:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cmp /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_36.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_36.export 2023-12-04T14:30:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_36.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_36.export 2023-12-04T14:30:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-04T14:30:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ image=image_37 2023-12-04T14:30:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ create_snap cluster2 mirror image_37 snap 2023-12-04T14:30:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:30:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:30:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-12-04T14:30:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-12-04T14:30:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:30:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-12-04T14:30:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:30:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local state=started 2023-12-04T14:30:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:30:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:30:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-12-04T14:30:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:30:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:30:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local test_state=started 2023-12-04T14:30:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_result 2023-12-04T14:30:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local current_state=stopped 2023-12-04T14:30:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2023-12-04T14:30:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_instance=cluster1 2023-12-04T14:30:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ grep -i state 2023-12-04T14:30:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1 2023-12-04T14:30:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance=cluster1 2023-12-04T14:30:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local loop_instance 2023-12-04T14:30:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-04T14:30:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sleep 0 2023-12-04T14:30:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-04T14:30:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ seq 0 3 2023-12-04T14:30:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-12-04T14:30:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T14:30:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T14:30:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T14:30:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T14:30:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T14:30:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-04T14:30:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:0 2023-12-04T14:30:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-04T14:30:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:0 2023-12-04T14:30:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T14:30:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-04T14:30:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=0 2023-12-04T14:30:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=0 2023-12-04T14:30:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-04T14:30:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-04T14:30:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-12-04T14:30:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-04T14:30:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2023-12-04T14:30:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T14:30:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T14:30:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T14:30:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T14:30:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T14:30:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-04T14:30:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:1 2023-12-04T14:30:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-04T14:30:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:1 2023-12-04T14:30:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T14:30:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-04T14:30:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=1 2023-12-04T14:30:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=1 2023-12-04T14:30:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-04T14:30:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-04T14:30:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2023-12-04T14:30:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-04T14:30:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2023-12-04T14:30:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T14:30:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T14:30:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T14:30:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T14:30:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T14:30:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-04T14:30:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:2 2023-12-04T14:30:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-04T14:30:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:2 2023-12-04T14:30:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T14:30:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-04T14:30:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=2 2023-12-04T14:30:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=2 2023-12-04T14:30:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-04T14:30:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-04T14:30:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2023-12-04T14:30:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-04T14:30:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd mirror status : get status for rbd mirror 2023-12-04T14:30:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:admin_socket: invalid command 2023-12-04T14:30:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T14:30:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_37 2023-12-04T14:30:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T14:30:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ shift 2023-12-04T14:30:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T14:30:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T14:30:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T14:30:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T14:30:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T14:30:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T14:30:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T14:30:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T14:30:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T14:30:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-04T14:30:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local local_cluster=cluster1:3 2023-12-04T14:30:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster=cluster1 2023-12-04T14:30:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance 2023-12-04T14:30:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-04T14:30:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local val=cluster1:3 2023-12-04T14:30:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local cluster_var_name=local_cluster 2023-12-04T14:30:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ local instance_var_name=instance 2023-12-04T14:30:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T14:30:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T14:30:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-04T14:30:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval local_cluster=cluster1 2023-12-04T14:30:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ local_cluster=cluster1 2023-12-04T14:30:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ eval instance=3 2023-12-04T14:30:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ instance=3 2023-12-04T14:30:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-04T14:30:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-04T14:30:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_37 2023-12-04T14:30:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ return 0 2023-12-04T14:30:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status_result=' "state": "Replaying"' 2023-12-04T14:30:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo ' "state": "Replaying"' 2023-12-04T14:30:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ grep -i Replaying 2023-12-04T14:30:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "state": "Replaying" 2023-12-04T14:30:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ current_state=started 2023-12-04T14:30:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test started = started 2023-12-04T14:30:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:30:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-12-04T14:30:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:30:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = journal ']' 2023-12-04T14:30:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' snapshot = snapshot ']' 2023-12-04T14:30:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-12-04T14:30:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_cluster=cluster1 2023-12-04T14:30:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:30:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster2-mirror-image_37.status 2023-12-04T14:30:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster2-mirror-image_37.status 2023-12-04T14:30:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local status_log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_37.status 2023-12-04T14:30:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ mkfname cluster1-mirror-image_37.status 2023-12-04T14:30:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo cluster1-mirror-image_37.status 2023-12-04T14:30:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-04T14:30:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local local_status_log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_37.status 2023-12-04T14:30:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-12-04T14:30:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:30:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:30:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:30:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-12-04T14:31:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Snapshot ID: 191 2023-12-04T14:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_37.status 2023-12-04T14:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T14:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_37.status 2023-12-04T14:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-12-04T14:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:31:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-04T14:31:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snapshot_id=191 2023-12-04T14:31:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:31:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:31:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.2 2023-12-04T14:31:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_37.status 2023-12-04T14:31:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:31:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:31:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:31:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_37.status 2023-12-04T14:31:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-04T14:31:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:31:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:31:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:31:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:31:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:31:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:31:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:31:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 191 = '' 2023-12-04T14:31:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:31:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.4 2023-12-04T14:31:00.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2023-12-04T14:31:00.785 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-04T14:31:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_37.status 2023-12-04T14:31:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:31:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:31:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:31:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_37.status 2023-12-04T14:31:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-04T14:31:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:31:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ true 2023-12-04T14:31:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:31:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:-:1.1: Document is empty 2023-12-04T14:31:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr: 2023-12-04T14:31:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:^ 2023-12-04T14:31:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id= 2023-12-04T14:31:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 191 = '' 2023-12-04T14:31:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-04T14:31:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 0.8 2023-12-04T14:31:01.189 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2023-12-04T14:31:01.189 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-12-04T14:31:01.189 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-12-04T14:31:01.190 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-12-04T14:31:01.190 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:31:01.189+0000 7f0535df0640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 86028) UID: 0 2023-12-04T14:31:01.191 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:31:01.189+0000 7f0535df0640 -1 osd.1 224 *** Got signal Terminated *** 2023-12-04T14:31:01.191 INFO:tasks.ceph.cluster2.osd.1.smithi176.stderr:2023-12-04T14:31:01.189+0000 7f0535df0640 -1 osd.1 224 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T14:31:01.191 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:31:01.190+0000 7f4c08575640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 86030) UID: 0 2023-12-04T14:31:01.191 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:31:01.190+0000 7f4c08575640 -1 osd.2 224 *** Got signal Terminated *** 2023-12-04T14:31:01.191 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:31:01.190+0000 7f4c08575640 -1 osd.2 224 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T14:31:01.192 INFO:tasks.ceph.cluster2.mon.a.smithi176.stderr:2023-12-04T14:31:01.190+0000 7f40037da640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 85720) UID: 0 2023-12-04T14:31:01.192 INFO:tasks.ceph.cluster2.mon.a.smithi176.stderr:2023-12-04T14:31:01.190+0000 7f40037da640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-04T14:31:01.192 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:31:01.190+0000 7f15d63b3640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 90669) UID: 0 2023-12-04T14:31:01.192 INFO:tasks.ceph.cluster2.mgr.x.smithi146.stderr:2023-12-04T14:31:01.190+0000 7f15d63b3640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-04T14:31:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_37.status 2023-12-04T14:31:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:31:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:31:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:31:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local log=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_37.status 2023-12-04T14:31:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-04T14:31:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-04T14:31:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-04T14:31:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local primary_snapshot_id=191 2023-12-04T14:31:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test 191 = 191 2023-12-04T14:31:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:31:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-12-04T14:31:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T14:31:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:31:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:31:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:31:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local s 2023-12-04T14:31:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-04T14:31:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ sleep 1 2023-12-04T14:31:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-12-04T14:31:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_37': 2023-12-04T14:31:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:31:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:31:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:31:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3bfecd311 2023-12-04T14:31:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3bfecd311 2023-12-04T14:31:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:31:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:31:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:31:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:31:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:57 2023 2023-12-04T14:31:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:23:57 2023 2023-12-04T14:31:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:57 2023 2023-12-04T14:31:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: protected: False 2023-12-04T14:31:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:31:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:31:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 573c9935-53e1-4562-b5e3-42d4b91a1ca2 2023-12-04T14:31:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:31:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 0 2023-12-04T14:31:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ compare_image_snaps mirror image_37 snap 2023-12-04T14:31:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pool=mirror 2023-12-04T14:31:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local image=image_37 2023-12-04T14:31:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local snap_name=snap 2023-12-04T14:31:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret=0 2023-12-04T14:31:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local rmt_export=/tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_37.export 2023-12-04T14:31:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local loc_export=/tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_37.export 2023-12-04T14:31:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -f /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_37.export /tmp/tmp.rDwoQpiP9p/cluster1-mirror-image_37.export 2023-12-04T14:31:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.rDwoQpiP9p/cluster2-mirror-image_37.export 2023-12-04T14:31:03.919 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:31:03.918+0000 7f4bfa770640 -1 reset not still connected to 0x56234b02af70 2023-12-04T14:31:03.919 INFO:tasks.ceph.cluster2.osd.2.smithi176.stderr:2023-12-04T14:31:03.918+0000 7f4bfa770640 -1 reset not still connected to 0x5623539a7380 2023-12-04T14:31:15.337 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2727395132280167) > weight (0.25) 2023-12-04T14:31:15.337 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.645428577535123) > weight (0.25) 2023-12-04T14:31:15.337 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2711712126099888) > weight (0.25) 2023-12-04T14:31:15.337 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.43929086638992265) > weight (0.25) 2023-12-04T14:31:15.337 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-12-04T14:32:10.755 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7237226218149422) > weight (0.25) 2023-12-04T14:32:10.755 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.37919428993994175) > weight (0.25) 2023-12-04T14:32:10.755 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-04T14:32:10.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-04T14:32:10.756 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-04T14:32:10.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr:2023-12-04T14:32:10.756+0000 7f7d4e4d1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 103931) UID: 1000 2023-12-04T14:32:15.757 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-04T14:32:15.757 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:32:35.339 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5065822429607466) > weight (0.25) 2023-12-04T14:32:35.339 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-04T14:32:35.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-04T14:32:35.340 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-04T14:32:35.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr:2023-12-04T14:32:35.340+0000 7f083ebbe640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 96361) UID: 1000 2023-12-04T14:32:36.341 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-04T14:32:36.341 INFO:teuthology.orchestra.run:waiting for 600 2023-12-04T14:34:38.979 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:34:38.976+0000 7f5183974640 -1 reset not still connected to 0x55e7baeb8750 2023-12-04T14:34:38.979 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:34:38.976+0000 7f5183974640 -1 reset not still connected to 0x55e7bb06b2b0 2023-12-04T14:34:38.980 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:34:38.976+0000 7f5183974640 -1 reset not still connected to 0x55e7bb4716c0 2023-12-04T14:34:38.980 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:34:38.976+0000 7f5183974640 -1 reset not still connected to 0x55e7bb471d40 2023-12-04T14:34:38.980 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:34:38.976+0000 7f5183974640 -1 reset not still connected to 0x55e7bb472d00 2023-12-04T14:34:38.980 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:34:38.976+0000 7f5183974640 -1 reset not still connected to 0x55e7bb4748f0 2023-12-04T14:34:38.980 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:34:38.976+0000 7f5183974640 -1 reset not still connected to 0x55e7bb474a90 2023-12-04T14:34:38.980 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:34:38.976+0000 7f5183974640 -1 reset not still connected to 0x55e7bc29e750 2023-12-04T14:34:38.980 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:34:38.976+0000 7f5183974640 -1 reset not still connected to 0x55e7bc29fba0 2023-12-04T14:36:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T14:36:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ cleanup 110 2023-12-04T14:36:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local error_code=110 2023-12-04T14:36:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ set +e 2023-12-04T14:36:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 110 -ne 0 ']' 2023-12-04T14:36:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ status 2023-12-04T14:36:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster daemon image_pool image_ns image 2023-12-04T14:36:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2023-12-04T14:36:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 status 2023-12-04T14:36:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 status' 2023-12-04T14:36:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:36:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster1 -s 2023-12-04T14:36:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: cluster: 2023-12-04T14:36:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 50899115-b691-4368-9c50-3c5017eb8b8a 2023-12-04T14:36:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: HEALTH_OK 2023-12-04T14:36:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: services: 2023-12-04T14:36:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mon: 1 daemons, quorum a (age 30m) 2023-12-04T14:36:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mgr: x(active, since 30m) 2023-12-04T14:36:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: osd: 3 osds: 3 up (since 30m), 3 in (since 30m) 2023-12-04T14:36:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: rbd-mirror: 4 daemons active (1 hosts) 2023-12-04T14:36:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: data: 2023-12-04T14:36:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: pools: 4 pools, 137 pgs 2023-12-04T14:36:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: objects: 1.37k objects, 4.6 GiB 2023-12-04T14:36:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: usage: 9.4 GiB used, 261 GiB / 270 GiB avail 2023-12-04T14:36:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: pgs: 137 active+clean 2023-12-04T14:36:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: io: 2023-12-04T14:36:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client: 1.2 KiB/s rd, 170 B/s wr, 1 op/s rd, 0 op/s wr 2023-12-04T14:36:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:36:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster1 service dump 2023-12-04T14:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:{ 2023-12-04T14:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "epoch": 30, 2023-12-04T14:36:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "modified": "2023-12-04T14:30:16.510315+0000", 2023-12-04T14:36:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "services": { 2023-12-04T14:36:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "rbd-mirror": { 2023-12-04T14:36:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "daemons": { 2023-12-04T14:36:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "summary": "", 2023-12-04T14:36:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "4258": { 2023-12-04T14:36:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "start_epoch": 6, 2023-12-04T14:36:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "start_stamp": "2023-12-04T14:07:43.358591+0000", 2023-12-04T14:36:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "gid": 4258, 2023-12-04T14:36:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "addr": "172.21.15.176:0/3756260804", 2023-12-04T14:36:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "metadata": { 2023-12-04T14:36:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "arch": "x86_64", 2023-12-04T14:36:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "ceph_release": "reef", 2023-12-04T14:36:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "ceph_version": "ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev)", 2023-12-04T14:36:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "ceph_version_short": "18.0.0-7672-g7c730b29", 2023-12-04T14:36:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-04T14:36:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "distro": "centos", 2023-12-04T14:36:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "distro_description": "CentOS Stream 9", 2023-12-04T14:36:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "distro_version": "9", 2023-12-04T14:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "hostname": "smithi176", 2023-12-04T14:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "id": "mirror.3", 2023-12-04T14:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "instance_id": "4258", 2023-12-04T14:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023", 2023-12-04T14:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "kernel_version": "5.14.0-388.el9.x86_64", 2023-12-04T14:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "mem_swap_kb": "0", 2023-12-04T14:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "mem_total_kb": "32488652", 2023-12-04T14:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "os": "Linux" 2023-12-04T14:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: }, 2023-12-04T14:36:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "task_status": {} 2023-12-04T14:36:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: }, 2023-12-04T14:36:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "4783": { 2023-12-04T14:36:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "start_epoch": 25, 2023-12-04T14:36:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "start_stamp": "2023-12-04T14:25:00.844966+0000", 2023-12-04T14:36:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "gid": 4783, 2023-12-04T14:36:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "addr": "172.21.15.176:0/2370326578", 2023-12-04T14:36:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "metadata": { 2023-12-04T14:36:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "arch": "x86_64", 2023-12-04T14:36:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "ceph_release": "reef", 2023-12-04T14:36:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "ceph_version": "ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev)", 2023-12-04T14:36:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "ceph_version_short": "18.0.0-7672-g7c730b29", 2023-12-04T14:36:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-04T14:36:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "distro": "centos", 2023-12-04T14:36:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "distro_description": "CentOS Stream 9", 2023-12-04T14:36:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "distro_version": "9", 2023-12-04T14:36:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "hostname": "smithi176", 2023-12-04T14:36:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "id": "mirror.2", 2023-12-04T14:36:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "instance_id": "4783", 2023-12-04T14:36:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023", 2023-12-04T14:36:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "kernel_version": "5.14.0-388.el9.x86_64", 2023-12-04T14:36:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "mem_swap_kb": "0", 2023-12-04T14:36:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "mem_total_kb": "32488652", 2023-12-04T14:36:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "os": "Linux" 2023-12-04T14:36:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: }, 2023-12-04T14:36:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "task_status": {} 2023-12-04T14:36:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: }, 2023-12-04T14:36:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "4881": { 2023-12-04T14:36:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "start_epoch": 28, 2023-12-04T14:36:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "start_stamp": "2023-12-04T14:27:19.848518+0000", 2023-12-04T14:36:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "gid": 4881, 2023-12-04T14:36:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "addr": "172.21.15.176:0/3596778951", 2023-12-04T14:36:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "metadata": { 2023-12-04T14:36:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "arch": "x86_64", 2023-12-04T14:36:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "ceph_release": "reef", 2023-12-04T14:36:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "ceph_version": "ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev)", 2023-12-04T14:36:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "ceph_version_short": "18.0.0-7672-g7c730b29", 2023-12-04T14:36:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-04T14:36:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "distro": "centos", 2023-12-04T14:36:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "distro_description": "CentOS Stream 9", 2023-12-04T14:36:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "distro_version": "9", 2023-12-04T14:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "hostname": "smithi176", 2023-12-04T14:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "id": "mirror.1", 2023-12-04T14:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "instance_id": "4881", 2023-12-04T14:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023", 2023-12-04T14:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "kernel_version": "5.14.0-388.el9.x86_64", 2023-12-04T14:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "mem_swap_kb": "0", 2023-12-04T14:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "mem_total_kb": "32488652", 2023-12-04T14:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "os": "Linux" 2023-12-04T14:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: }, 2023-12-04T14:36:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "task_status": {} 2023-12-04T14:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: }, 2023-12-04T14:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "5326": { 2023-12-04T14:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "start_epoch": 30, 2023-12-04T14:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "start_stamp": "2023-12-04T14:30:14.834670+0000", 2023-12-04T14:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "gid": 5326, 2023-12-04T14:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "addr": "172.21.15.176:0/335391891", 2023-12-04T14:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "metadata": { 2023-12-04T14:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "arch": "x86_64", 2023-12-04T14:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "ceph_release": "reef", 2023-12-04T14:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "ceph_version": "ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev)", 2023-12-04T14:36:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "ceph_version_short": "18.0.0-7672-g7c730b29", 2023-12-04T14:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-12-04T14:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "distro": "centos", 2023-12-04T14:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "distro_description": "CentOS Stream 9", 2023-12-04T14:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "distro_version": "9", 2023-12-04T14:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "hostname": "smithi176", 2023-12-04T14:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "id": "mirror.0", 2023-12-04T14:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "instance_id": "5326", 2023-12-04T14:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023", 2023-12-04T14:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "kernel_version": "5.14.0-388.el9.x86_64", 2023-12-04T14:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "mem_swap_kb": "0", 2023-12-04T14:36:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "mem_total_kb": "32488652", 2023-12-04T14:36:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "os": "Linux" 2023-12-04T14:36:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: }, 2023-12-04T14:36:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "task_status": {} 2023-12-04T14:36:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: } 2023-12-04T14:36:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: } 2023-12-04T14:36:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: } 2023-12-04T14:36:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: } 2023-12-04T14:36:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:} 2023-12-04T14:36:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:36:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster1 service status 2023-12-04T14:36:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:{ 2023-12-04T14:36:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "rbd-mirror": { 2023-12-04T14:36:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "4258": { 2023-12-04T14:36:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "status_stamp": "2023-12-04T14:35:48.441423+0000", 2023-12-04T14:36:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "last_beacon": "2023-12-04T14:36:03.442154+0000", 2023-12-04T14:36:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "status": { 2023-12-04T14:36:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":50,\"image_error_count\":0,\"image_local_count\":50,\"image_remote_count\":50,\"image_warning_count\":0,\"instance_id\":\"4307\",\"leader\":true,\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0,\"instance_id\":\"4310\",\"leader\":true}}" 2023-12-04T14:36:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: } 2023-12-04T14:36:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: }, 2023-12-04T14:36:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "4783": { 2023-12-04T14:36:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "status_stamp": "2023-12-04T14:35:35.869483+0000", 2023-12-04T14:36:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "last_beacon": "2023-12-04T14:36:00.870330+0000", 2023-12-04T14:36:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "status": { 2023-12-04T14:36:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4786\"},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4790\"}}" 2023-12-04T14:36:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: } 2023-12-04T14:36:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: }, 2023-12-04T14:36:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "4881": { 2023-12-04T14:36:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "status_stamp": "2023-12-04T14:35:54.867392+0000", 2023-12-04T14:36:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "last_beacon": "2023-12-04T14:35:59.867532+0000", 2023-12-04T14:36:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "status": { 2023-12-04T14:36:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4884\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4890\"}}" 2023-12-04T14:36:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: } 2023-12-04T14:36:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: }, 2023-12-04T14:36:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "5326": { 2023-12-04T14:36:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "status_stamp": "2023-12-04T14:35:49.847947+0000", 2023-12-04T14:36:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "last_beacon": "2023-12-04T14:35:59.848194+0000", 2023-12-04T14:36:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "status": { 2023-12-04T14:36:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5329\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5331\"}}" 2023-12-04T14:36:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: } 2023-12-04T14:36:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: } 2023-12-04T14:36:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: } 2023-12-04T14:36:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:} 2023-12-04T14:36:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-04T14:36:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-04T14:36:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror images 2023-12-04T14:36:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror images' 2023-12-04T14:36:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' ls -l 2023-12-04T14:36:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:NAME SIZE PARENT FMT PROT LOCK 2023-12-04T14:36:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_1 128 MiB 2 2023-12-04T14:36:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_1@snap 128 MiB 2 2023-12-04T14:36:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10 128 MiB 2 2023-12-04T14:36:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10@snap 128 MiB 2 2023-12-04T14:36:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_11 128 MiB 2 2023-12-04T14:36:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_11@snap 128 MiB 2 2023-12-04T14:36:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_12 128 MiB 2 2023-12-04T14:36:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_12@snap 128 MiB 2 2023-12-04T14:36:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_13 128 MiB 2 2023-12-04T14:36:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_13@snap 128 MiB 2 2023-12-04T14:36:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_14 128 MiB 2 2023-12-04T14:36:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_14@snap 128 MiB 2 2023-12-04T14:36:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_15 128 MiB 2 2023-12-04T14:36:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_15@snap 128 MiB 2 2023-12-04T14:36:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_16 128 MiB 2 2023-12-04T14:36:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_16@snap 128 MiB 2 2023-12-04T14:36:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_17 128 MiB 2 2023-12-04T14:36:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_17@snap 128 MiB 2 2023-12-04T14:36:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_18 128 MiB 2 2023-12-04T14:36:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_18@snap 128 MiB 2 2023-12-04T14:36:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_19 128 MiB 2 2023-12-04T14:36:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_19@snap 128 MiB 2 2023-12-04T14:36:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_2 128 MiB 2 2023-12-04T14:36:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_2@snap 128 MiB 2 2023-12-04T14:36:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_20 128 MiB 2 2023-12-04T14:36:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_20@snap 128 MiB 2 2023-12-04T14:36:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_21 128 MiB 2 2023-12-04T14:36:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_21@snap 128 MiB 2 2023-12-04T14:36:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_22 128 MiB 2 2023-12-04T14:36:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_22@snap 128 MiB 2 2023-12-04T14:36:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_23 128 MiB 2 2023-12-04T14:36:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_23@snap 128 MiB 2 2023-12-04T14:36:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_24 128 MiB 2 2023-12-04T14:36:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_24@snap 128 MiB 2 2023-12-04T14:36:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_25 128 MiB 2 2023-12-04T14:36:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_25@snap 128 MiB 2 2023-12-04T14:36:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_26 128 MiB 2 2023-12-04T14:36:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_26@snap 128 MiB 2 2023-12-04T14:36:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_27 128 MiB 2 2023-12-04T14:36:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_27@snap 128 MiB 2 2023-12-04T14:36:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_28 128 MiB 2 2023-12-04T14:36:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_28@snap 128 MiB 2 2023-12-04T14:36:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_29 128 MiB 2 2023-12-04T14:36:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_29@snap 128 MiB 2 2023-12-04T14:36:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_3 128 MiB 2 2023-12-04T14:36:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_3@snap 128 MiB 2 2023-12-04T14:36:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_30 128 MiB 2 2023-12-04T14:36:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_30@snap 128 MiB 2 2023-12-04T14:36:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_31 128 MiB 2 2023-12-04T14:36:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_31@snap 128 MiB 2 2023-12-04T14:36:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_32 128 MiB 2 2023-12-04T14:36:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_32@snap 128 MiB 2 2023-12-04T14:36:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_33 128 MiB 2 2023-12-04T14:36:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_33@snap 128 MiB 2 2023-12-04T14:36:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_34 128 MiB 2 2023-12-04T14:36:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_34@snap 128 MiB 2 2023-12-04T14:36:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_35 128 MiB 2 2023-12-04T14:36:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_35@snap 128 MiB 2 2023-12-04T14:36:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_36 128 MiB 2 2023-12-04T14:36:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_36@snap 128 MiB 2 2023-12-04T14:36:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_37 128 MiB 2 2023-12-04T14:36:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_37@snap 128 MiB 2 2023-12-04T14:36:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_38 128 MiB 2 2023-12-04T14:36:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_39 128 MiB 2 2023-12-04T14:36:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_4 128 MiB 2 2023-12-04T14:36:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_4@snap 128 MiB 2 2023-12-04T14:36:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_40 128 MiB 2 2023-12-04T14:36:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_41 128 MiB 2 2023-12-04T14:36:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_42 128 MiB 2 2023-12-04T14:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_43 128 MiB 2 2023-12-04T14:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_44 128 MiB 2 2023-12-04T14:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_45 128 MiB 2 2023-12-04T14:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_46 128 MiB 2 2023-12-04T14:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_47 128 MiB 2 2023-12-04T14:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_48 128 MiB 2 2023-12-04T14:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_49 128 MiB 2 2023-12-04T14:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_5 128 MiB 2 2023-12-04T14:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_5@snap 128 MiB 2 2023-12-04T14:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_50 128 MiB 2 2023-12-04T14:36:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_6 128 MiB 2 2023-12-04T14:36:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_6@snap 128 MiB 2 2023-12-04T14:36:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_7 128 MiB 2 2023-12-04T14:36:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_7@snap 128 MiB 2 2023-12-04T14:36:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_8 128 MiB 2 2023-12-04T14:36:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_8@snap 128 MiB 2 2023-12-04T14:36:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_9 128 MiB 2 2023-12-04T14:36:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_9@snap 128 MiB 2 2023-12-04T14:36:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror mirror pool info 2023-12-04T14:36:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror mirror pool info' 2023-12-04T14:36:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool info 2023-12-04T14:36:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mode: image 2023-12-04T14:36:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Site Name: cluster1 2023-12-04T14:36:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Peer Sites: 2023-12-04T14:36:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:UUID: 4d244b58-7457-4b89-a2f7-d69add210e3b 2023-12-04T14:36:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Name: cluster2 2023-12-04T14:36:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirror UUID: f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Direction: rx-tx 2023-12-04T14:36:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Client: client.mirror 2023-12-04T14:36:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror mirror pool status 2023-12-04T14:36:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror mirror pool status' 2023-12-04T14:36:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:36:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool status --verbose 2023-12-04T14:36:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:health: WARNING 2023-12-04T14:36:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:daemon health: OK 2023-12-04T14:36:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image health: WARNING 2023-12-04T14:36:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:images: 50 total 2023-12-04T14:36:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 50 starting_replay 2023-12-04T14:36:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:DAEMONS 2023-12-04T14:36:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 4258: 2023-12-04T14:36:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 4307 2023-12-04T14:36:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.3 2023-12-04T14:36:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: true 2023-12-04T14:36:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 4783: 2023-12-04T14:36:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 4786 2023-12-04T14:36:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.2 2023-12-04T14:36:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: false 2023-12-04T14:36:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 4881: 2023-12-04T14:36:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 4884 2023-12-04T14:36:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.1 2023-12-04T14:36:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: false 2023-12-04T14:36:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 5326: 2023-12-04T14:36:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 5329 2023-12-04T14:36:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.0 2023-12-04T14:36:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: false 2023-12-04T14:36:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:IMAGES 2023-12-04T14:36:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_1: 2023-12-04T14:36:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: b928065e-ca9b-41a2-9e4f-db3d8351e631 2023-12-04T14:36:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133996544.0,"last_snapshot_bytes":133996544,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700016,"remote_snapshot_timestamp":1701700016,"replay_state":"idle"} 2023-12-04T14:36:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10: 2023-12-04T14:36:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 0d9ffbfe-5f0b-4261-82aa-d3c625dda130 2023-12-04T14:36:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66712576.0,"last_snapshot_bytes":133425152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700078,"remote_snapshot_timestamp":1701700078,"replay_state":"idle"} 2023-12-04T14:36:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_11: 2023-12-04T14:36:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 99047729-2de2-4c91-90dc-1e2bfe4f2b5f 2023-12-04T14:36:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66881024.0,"last_snapshot_bytes":133762048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700084,"remote_snapshot_timestamp":1701700084,"replay_state":"idle"} 2023-12-04T14:36:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_12: 2023-12-04T14:36:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 5f322e9d-b400-4d44-81e0-ffdd216398b2 2023-12-04T14:36:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66597376.0,"last_snapshot_bytes":133194752,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700090,"remote_snapshot_timestamp":1701700090,"replay_state":"idle"} 2023-12-04T14:36:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_13: 2023-12-04T14:36:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: ac478d96-184c-4eaa-8c62-f6f0e8288dff 2023-12-04T14:36:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66763264.0,"last_snapshot_bytes":133526528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700096,"remote_snapshot_timestamp":1701700096,"replay_state":"idle"} 2023-12-04T14:36:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_14: 2023-12-04T14:36:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 8794a2f7-791e-4d63-afdb-a1f297b316fa 2023-12-04T14:36:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67013120.0,"last_snapshot_bytes":134026240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700102,"remote_snapshot_timestamp":1701700102,"replay_state":"idle"} 2023-12-04T14:36:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_15: 2023-12-04T14:36:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 0e39536e-cf2c-4b79-ae6d-a5d1ede7920c 2023-12-04T14:36:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67047936.0,"last_snapshot_bytes":134095872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700109,"remote_snapshot_timestamp":1701700109,"replay_state":"idle"} 2023-12-04T14:36:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_16: 2023-12-04T14:36:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 5854cc13-6872-45e6-8c82-645c028f7199 2023-12-04T14:36:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66460160.0,"last_snapshot_bytes":132920320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700117,"remote_snapshot_timestamp":1701700117,"replay_state":"idle"} 2023-12-04T14:36:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_17: 2023-12-04T14:36:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 7bfe12f9-a769-4f59-bca6-8c32a93f3809 2023-12-04T14:36:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66864640.0,"last_snapshot_bytes":133729280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700123,"remote_snapshot_timestamp":1701700123,"replay_state":"idle"} 2023-12-04T14:36:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_18: 2023-12-04T14:36:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 9d431cf5-1ffc-4498-b3e0-5bba35366d30 2023-12-04T14:36:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67021312.0,"last_snapshot_bytes":134042624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700131,"remote_snapshot_timestamp":1701700131,"replay_state":"idle"} 2023-12-04T14:36:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_19: 2023-12-04T14:36:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 44e764dc-83c2-44b2-bca6-672e14a04ca5 2023-12-04T14:36:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65781248.0,"last_snapshot_bytes":131562496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700138,"remote_snapshot_timestamp":1701700138,"replay_state":"idle"} 2023-12-04T14:36:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_2: 2023-12-04T14:36:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 9fe30a7e-3975-4be8-9ddc-ffdbf8f787e1 2023-12-04T14:36:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133588992.0,"last_snapshot_bytes":133588992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700022,"remote_snapshot_timestamp":1701700022,"replay_state":"idle"} 2023-12-04T14:36:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_20: 2023-12-04T14:36:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: e7fd68b0-bd04-4c3c-af90-fd5517fef987 2023-12-04T14:36:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65917952.0,"last_snapshot_bytes":131835904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700146,"remote_snapshot_timestamp":1701700146,"replay_state":"idle"} 2023-12-04T14:36:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_21: 2023-12-04T14:36:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 2a201e15-51f0-492a-8b9c-28ef7a490beb 2023-12-04T14:36:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66464768.0,"last_snapshot_bytes":132929536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700153,"remote_snapshot_timestamp":1701700153,"replay_state":"idle"} 2023-12-04T14:36:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_22: 2023-12-04T14:36:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 059e9df9-d50d-4eb9-8306-2dcc0c369e9c 2023-12-04T14:36:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67096576.0,"last_snapshot_bytes":134193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700160,"remote_snapshot_timestamp":1701700160,"replay_state":"idle"} 2023-12-04T14:36:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_23: 2023-12-04T14:36:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: e304857c-103b-447a-ae76-3be95494c2ac 2023-12-04T14:36:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67072000.0,"last_snapshot_bytes":134144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700168,"remote_snapshot_timestamp":1701700168,"replay_state":"idle"} 2023-12-04T14:36:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_24: 2023-12-04T14:36:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: eb8df2f9-471f-4269-a2e3-70777410572c 2023-12-04T14:36:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67077120.0,"last_snapshot_bytes":134154240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700174,"remote_snapshot_timestamp":1701700174,"replay_state":"idle"} 2023-12-04T14:36:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_25: 2023-12-04T14:36:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 44871cb2-b60d-4e0e-8e66-9d9bdcbae254 2023-12-04T14:36:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67048448.0,"last_snapshot_bytes":134096896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700182,"remote_snapshot_timestamp":1701700182,"replay_state":"idle"} 2023-12-04T14:36:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_26: 2023-12-04T14:36:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: edd4af1f-3c2b-4169-adc5-5d59c2cc5a61 2023-12-04T14:36:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67077120.0,"last_snapshot_bytes":134154240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700188,"remote_snapshot_timestamp":1701700188,"replay_state":"idle"} 2023-12-04T14:36:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_27: 2023-12-04T14:36:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: e393e7c2-3dbe-4e5b-9517-da1c79d5fe47 2023-12-04T14:36:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66303488.0,"last_snapshot_bytes":132606976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700194,"remote_snapshot_timestamp":1701700194,"replay_state":"idle"} 2023-12-04T14:36:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_28: 2023-12-04T14:36:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: ad2b0660-9c28-48f0-8b26-08f4860ad09e 2023-12-04T14:36:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67006464.0,"last_snapshot_bytes":134012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700200,"remote_snapshot_timestamp":1701700200,"replay_state":"idle"} 2023-12-04T14:36:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_29: 2023-12-04T14:36:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 040b1033-d414-4ab1-bb84-83c330f7ff65 2023-12-04T14:36:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66017792.0,"last_snapshot_bytes":132035584,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700206,"remote_snapshot_timestamp":1701700206,"replay_state":"idle"} 2023-12-04T14:36:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_3: 2023-12-04T14:36:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: d5429052-81ab-46d6-ab3d-16695051435d 2023-12-04T14:36:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67056640.0,"last_snapshot_bytes":134113280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700029,"remote_snapshot_timestamp":1701700029,"replay_state":"idle"} 2023-12-04T14:36:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_30: 2023-12-04T14:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 233226c3-03eb-48bb-bc7e-fe8f5e6be5a9 2023-12-04T14:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66885632.0,"last_snapshot_bytes":133771264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700212,"remote_snapshot_timestamp":1701700212,"replay_state":"idle"} 2023-12-04T14:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_31: 2023-12-04T14:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 55aa29aa-3f60-443f-a765-b4d1a2257a41 2023-12-04T14:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66879488.0,"last_snapshot_bytes":133758976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700220,"remote_snapshot_timestamp":1701700220,"replay_state":"idle"} 2023-12-04T14:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_32: 2023-12-04T14:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: da0438b0-0666-4c5d-bca2-497b527c3c8a 2023-12-04T14:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65926656.0,"last_snapshot_bytes":131853312,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700228,"remote_snapshot_timestamp":1701700228,"replay_state":"idle"} 2023-12-04T14:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_33: 2023-12-04T14:36:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 709187ee-c3f5-47e9-a97c-a08560690fe3 2023-12-04T14:36:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67107840.0,"last_snapshot_bytes":134215680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700234,"remote_snapshot_timestamp":1701700234,"replay_state":"idle"} 2023-12-04T14:36:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_34: 2023-12-04T14:36:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 53abed9b-8d5d-4a15-a122-712b8dc9c3d5 2023-12-04T14:36:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67043328.0,"last_snapshot_bytes":134086656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700242,"remote_snapshot_timestamp":1701700242,"replay_state":"idle"} 2023-12-04T14:36:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_35: 2023-12-04T14:36:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 75dc1148-0c6b-4bf4-962c-9f4dc15daa97 2023-12-04T14:36:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66454016.0,"last_snapshot_bytes":132908032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700248,"remote_snapshot_timestamp":1701700248,"replay_state":"idle"} 2023-12-04T14:36:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_36: 2023-12-04T14:36:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: c88f1ade-8455-43f9-be6e-45b69b6f15a1 2023-12-04T14:36:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67024896.0,"last_snapshot_bytes":134049792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700254,"remote_snapshot_timestamp":1701700254,"replay_state":"idle"} 2023-12-04T14:36:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_37: 2023-12-04T14:36:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 573c9935-53e1-4562-b5e3-42d4b91a1ca2 2023-12-04T14:36:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66155008.0,"last_snapshot_bytes":132310016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700260,"remote_snapshot_timestamp":1701700260,"replay_state":"idle"} 2023-12-04T14:36:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_38: 2023-12-04T14:36:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: f19d0e0b-d80d-47ea-9448-f5c84f9d4cc6 2023-12-04T14:36:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699844,"remote_snapshot_timestamp":1701699844,"replay_state":"idle"} 2023-12-04T14:36:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_39: 2023-12-04T14:36:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 622dfb37-1a24-4c28-8ac0-9f49b29eeead 2023-12-04T14:36:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699852,"remote_snapshot_timestamp":1701699852,"replay_state":"idle"} 2023-12-04T14:36:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_4: 2023-12-04T14:36:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 60498ab7-7310-4365-aa39-30c7019acf9f 2023-12-04T14:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66899968.0,"last_snapshot_bytes":133799936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700036,"remote_snapshot_timestamp":1701700036,"replay_state":"idle"} 2023-12-04T14:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_40: 2023-12-04T14:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: f1ae4816-50e9-48b7-a351-3bb0cf0cd5bb 2023-12-04T14:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699867,"remote_snapshot_timestamp":1701699867,"replay_state":"idle"} 2023-12-04T14:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_41: 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 4985d9fc-9c9b-4cb0-a17e-3e72f3e74408 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699874,"remote_snapshot_timestamp":1701699874,"replay_state":"idle"} 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_42: 2023-12-04T14:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: fe0ec733-207f-4160-afad-a0d4633cee01 2023-12-04T14:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699882,"remote_snapshot_timestamp":1701699882,"replay_state":"idle"} 2023-12-04T14:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_43: 2023-12-04T14:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 5a31e773-c4a1-4a79-9d13-74cc228e5bc2 2023-12-04T14:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699894,"remote_snapshot_timestamp":1701699894,"replay_state":"idle"} 2023-12-04T14:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_44: 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 6705f07a-ea33-4f0a-b886-387067319889 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699907,"remote_snapshot_timestamp":1701699907,"replay_state":"idle"} 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_45: 2023-12-04T14:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: bd2f1aeb-d6bf-4777-b4de-b7dbba39dc7b 2023-12-04T14:36:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699915,"remote_snapshot_timestamp":1701699915,"replay_state":"idle"} 2023-12-04T14:36:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_46: 2023-12-04T14:36:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: f4eea075-92cf-457e-8d7a-77a594f0d72d 2023-12-04T14:36:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699923,"remote_snapshot_timestamp":1701699923,"replay_state":"idle"} 2023-12-04T14:36:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_47: 2023-12-04T14:36:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 1c9b05a7-06ae-420c-ab01-732da45bf605 2023-12-04T14:36:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699937,"remote_snapshot_timestamp":1701699937,"replay_state":"idle"} 2023-12-04T14:36:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_48: 2023-12-04T14:36:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 75662a66-3ee6-422c-bc5b-2ed60a3a9402 2023-12-04T14:36:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699952,"remote_snapshot_timestamp":1701699952,"replay_state":"idle"} 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_49: 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: c368a0ff-abe8-4e7e-bf74-d33daf9fed3d 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699963,"remote_snapshot_timestamp":1701699963,"replay_state":"idle"} 2023-12-04T14:36:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_5: 2023-12-04T14:36:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 90bc0c25-b4ac-4b4a-8f12-bbe6c286b2cd 2023-12-04T14:36:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66523648.0,"last_snapshot_bytes":133047296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700043,"remote_snapshot_timestamp":1701700043,"replay_state":"idle"} 2023-12-04T14:36:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_50: 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: b6c8de6f-5527-43d7-be14-65c94f60c736 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701699969,"remote_snapshot_timestamp":1701699969,"replay_state":"idle"} 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_6: 2023-12-04T14:36:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 481d8ed8-0c1d-4f88-8ce7-bbcfd2363766 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66422784.0,"last_snapshot_bytes":132845568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700050,"remote_snapshot_timestamp":1701700050,"replay_state":"idle"} 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_7: 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 431573ac-8014-4867-b7be-738ee54b8f15 2023-12-04T14:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65942016.0,"last_snapshot_bytes":131884032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700058,"remote_snapshot_timestamp":1701700058,"replay_state":"idle"} 2023-12-04T14:36:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_8: 2023-12-04T14:36:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: 14168f82-d9d2-41fc-99ed-f1205d8ef156 2023-12-04T14:36:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67040768.0,"last_snapshot_bytes":134081536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700064,"remote_snapshot_timestamp":1701700064,"replay_state":"idle"} 2023-12-04T14:36:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_9: 2023-12-04T14:36:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: global_id: fc8c6534-5c92-48ae-b43e-099c17bc6417 2023-12-04T14:36:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+replaying 2023-12-04T14:36:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67107840.0,"last_snapshot_bytes":134215680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1701700071,"remote_snapshot_timestamp":1701700071,"replay_state":"idle"} 2023-12-04T14:36:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: service: mirror.3 on smithi176 2023-12-04T14:36:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:43 2023-12-04T14:36:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: peer_sites: 2023-12-04T14:36:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: name: cluster2 2023-12-04T14:36:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: state: up+starting_replay 2023-12-04T14:36:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: description: starting replay 2023-12-04T14:36:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: last_update: 2023-12-04 14:35:35 2023-12-04T14:36:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' ls 2023-12-04T14:36:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_1 info 2023-12-04T14:36:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_1 info' 2023-12-04T14:36:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_1 2023-12-04T14:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_1': 2023-12-04T14:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 11a246f45543 2023-12-04T14:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.11a246f45543 2023-12-04T14:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:17:32 2023 2023-12-04T14:36:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:26:59 2023 2023-12-04T14:36:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:17:32 2023 2023-12-04T14:36:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: b928065e-ca9b-41a2-9e4f-db3d8351e631 2023-12-04T14:36:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_1 journal status 2023-12-04T14:36:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_1 journal status' 2023-12-04T14:36:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_1 2023-12-04T14:36:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_1 2023-12-04T14:36:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_1 snapshots' 2023-12-04T14:36:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_1 snapshots 2023-12-04T14:36:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_1 2023-12-04T14:36:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 118 snap 128 MiB Mon Dec 4 14:26:56 2023 user 2023-12-04T14:36:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 119 .mirror.non_primary.b928065e-ca9b-41a2-9e4f-db3d8351e631.d45ff1a3-b366-4ae4-ba15-e3a3d64168cc 128 MiB Mon Dec 4 14:26:57 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:119 copied) 2023-12-04T14:36:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_10 info 2023-12-04T14:36:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_10 info' 2023-12-04T14:36:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_10 2023-12-04T14:36:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_10': 2023-12-04T14:36:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3fd89eee1 2023-12-04T14:36:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3fd89eee1 2023-12-04T14:36:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:06 2023 2023-12-04T14:36:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:28:00 2023 2023-12-04T14:36:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:06 2023 2023-12-04T14:36:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 0d9ffbfe-5f0b-4261-82aa-d3c625dda130 2023-12-04T14:36:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_10 journal status 2023-12-04T14:36:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_10 journal status' 2023-12-04T14:36:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_10 2023-12-04T14:36:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_10 2023-12-04T14:36:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_10 snapshots 2023-12-04T14:36:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_10 snapshots' 2023-12-04T14:36:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_10 2023-12-04T14:36:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 145 snap 128 MiB Mon Dec 4 14:27:58 2023 user 2023-12-04T14:36:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 146 .mirror.non_primary.0d9ffbfe-5f0b-4261-82aa-d3c625dda130.45749777-9163-4990-980a-49a8a545809d 128 MiB Mon Dec 4 14:27:59 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:137 copied) 2023-12-04T14:36:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_11 info' 2023-12-04T14:36:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_11 info 2023-12-04T14:36:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_11 2023-12-04T14:36:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_11': 2023-12-04T14:36:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d32b04c6bd 2023-12-04T14:36:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d32b04c6bd 2023-12-04T14:36:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:15 2023 2023-12-04T14:36:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:28:07 2023 2023-12-04T14:36:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:15 2023 2023-12-04T14:36:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 99047729-2de2-4c91-90dc-1e2bfe4f2b5f 2023-12-04T14:36:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_11 journal status 2023-12-04T14:36:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_11 journal status' 2023-12-04T14:36:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_11 2023-12-04T14:36:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_11 2023-12-04T14:36:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_11 snapshots 2023-12-04T14:36:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_11 snapshots' 2023-12-04T14:36:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_11 2023-12-04T14:36:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 148 snap 128 MiB Mon Dec 4 14:28:04 2023 user 2023-12-04T14:36:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 149 .mirror.non_primary.99047729-2de2-4c91-90dc-1e2bfe4f2b5f.24bbd38d-58b3-4acf-a09f-e6e2006664f3 128 MiB Mon Dec 4 14:28:05 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:139 copied) 2023-12-04T14:36:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_12 info 2023-12-04T14:36:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_12 info' 2023-12-04T14:36:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_12 2023-12-04T14:36:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_12': 2023-12-04T14:36:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3719168a4 2023-12-04T14:36:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3719168a4 2023-12-04T14:36:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:25 2023 2023-12-04T14:36:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:28:13 2023 2023-12-04T14:36:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:25 2023 2023-12-04T14:36:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 5f322e9d-b400-4d44-81e0-ffdd216398b2 2023-12-04T14:36:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_12 journal status 2023-12-04T14:36:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_12 journal status' 2023-12-04T14:36:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_12 2023-12-04T14:36:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_12 2023-12-04T14:36:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_12 snapshots 2023-12-04T14:36:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_12 snapshots' 2023-12-04T14:36:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_12 2023-12-04T14:36:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 151 snap 128 MiB Mon Dec 4 14:28:10 2023 user 2023-12-04T14:36:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 152 .mirror.non_primary.5f322e9d-b400-4d44-81e0-ffdd216398b2.ea36d830-6152-4e0e-a187-7bc7cc9a037e 128 MiB Mon Dec 4 14:28:11 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:141 copied) 2023-12-04T14:36:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_13 info' 2023-12-04T14:36:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_13 info 2023-12-04T14:36:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_13 2023-12-04T14:36:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_13': 2023-12-04T14:36:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d34174832a 2023-12-04T14:36:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d34174832a 2023-12-04T14:36:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:37 2023 2023-12-04T14:36:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:28:19 2023 2023-12-04T14:36:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:37 2023 2023-12-04T14:36:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: ac478d96-184c-4eaa-8c62-f6f0e8288dff 2023-12-04T14:36:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_13 journal status 2023-12-04T14:36:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_13 journal status' 2023-12-04T14:36:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_13 2023-12-04T14:36:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_13 2023-12-04T14:36:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_13 snapshots 2023-12-04T14:36:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_13 snapshots' 2023-12-04T14:36:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_13 2023-12-04T14:36:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 154 snap 128 MiB Mon Dec 4 14:28:16 2023 user 2023-12-04T14:36:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 155 .mirror.non_primary.ac478d96-184c-4eaa-8c62-f6f0e8288dff.12888f20-108c-4a9c-b988-25d6f3a39551 128 MiB Mon Dec 4 14:28:17 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:143 copied) 2023-12-04T14:36:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_14 info 2023-12-04T14:36:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_14 info' 2023-12-04T14:36:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_14 2023-12-04T14:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_14': 2023-12-04T14:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3f464ab05 2023-12-04T14:36:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3f464ab05 2023-12-04T14:36:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:19:45 2023 2023-12-04T14:36:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:28:26 2023 2023-12-04T14:36:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:19:45 2023 2023-12-04T14:36:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 8794a2f7-791e-4d63-afdb-a1f297b316fa 2023-12-04T14:36:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_14 journal status 2023-12-04T14:36:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_14 journal status' 2023-12-04T14:36:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_14 2023-12-04T14:36:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_14 2023-12-04T14:36:05.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_14 snapshots 2023-12-04T14:36:05.464 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_14 snapshots' 2023-12-04T14:36:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_14 2023-12-04T14:36:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 157 snap 128 MiB Mon Dec 4 14:28:23 2023 user 2023-12-04T14:36:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 158 .mirror.non_primary.8794a2f7-791e-4d63-afdb-a1f297b316fa.a62fd872-741f-49f2-8b16-d0174d9e94c6 128 MiB Mon Dec 4 14:28:24 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:145 copied) 2023-12-04T14:36:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_15 info' 2023-12-04T14:36:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_15 info 2023-12-04T14:36:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_15 2023-12-04T14:36:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_15': 2023-12-04T14:36:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3b3ad0417 2023-12-04T14:36:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3b3ad0417 2023-12-04T14:36:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:00 2023 2023-12-04T14:36:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:28:33 2023 2023-12-04T14:36:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:00 2023 2023-12-04T14:36:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 0e39536e-cf2c-4b79-ae6d-a5d1ede7920c 2023-12-04T14:36:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_15 journal status' 2023-12-04T14:36:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_15 journal status 2023-12-04T14:36:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_15 2023-12-04T14:36:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_15 2023-12-04T14:36:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_15 snapshots 2023-12-04T14:36:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_15 snapshots' 2023-12-04T14:36:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_15 2023-12-04T14:36:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 160 snap 128 MiB Mon Dec 4 14:28:30 2023 user 2023-12-04T14:36:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 161 .mirror.non_primary.0e39536e-cf2c-4b79-ae6d-a5d1ede7920c.b8928e56-4fd6-45ae-9c8e-081112568e12 128 MiB Mon Dec 4 14:28:31 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:147 copied) 2023-12-04T14:36:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_16 info 2023-12-04T14:36:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_16 info' 2023-12-04T14:36:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_16 2023-12-04T14:36:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_16': 2023-12-04T14:36:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3ea48df64 2023-12-04T14:36:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3ea48df64 2023-12-04T14:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:08 2023 2023-12-04T14:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:28:39 2023 2023-12-04T14:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:08 2023 2023-12-04T14:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 5854cc13-6872-45e6-8c82-645c028f7199 2023-12-04T14:36:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_16 journal status 2023-12-04T14:36:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_16 journal status' 2023-12-04T14:36:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_16 2023-12-04T14:36:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_16 2023-12-04T14:36:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_16 snapshots 2023-12-04T14:36:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_16 snapshots' 2023-12-04T14:36:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_16 2023-12-04T14:36:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 163 snap 128 MiB Mon Dec 4 14:28:37 2023 user 2023-12-04T14:36:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 164 .mirror.non_primary.5854cc13-6872-45e6-8c82-645c028f7199.bfd534c8-2751-4828-8983-aace8d5b0419 128 MiB Mon Dec 4 14:28:38 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:149 copied) 2023-12-04T14:36:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_17 info 2023-12-04T14:36:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_17 info' 2023-12-04T14:36:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_17 2023-12-04T14:36:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_17': 2023-12-04T14:36:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3f05e87aa 2023-12-04T14:36:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3f05e87aa 2023-12-04T14:36:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:21 2023 2023-12-04T14:36:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:28:47 2023 2023-12-04T14:36:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:21 2023 2023-12-04T14:36:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 7bfe12f9-a769-4f59-bca6-8c32a93f3809 2023-12-04T14:36:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_17 journal status 2023-12-04T14:36:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_17 journal status' 2023-12-04T14:36:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_17 2023-12-04T14:36:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_17 2023-12-04T14:36:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_17 snapshots 2023-12-04T14:36:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_17 snapshots' 2023-12-04T14:36:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_17 2023-12-04T14:36:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 166 snap 128 MiB Mon Dec 4 14:28:44 2023 user 2023-12-04T14:36:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 167 .mirror.non_primary.7bfe12f9-a769-4f59-bca6-8c32a93f3809.ed1107b1-ac76-4558-88d2-92ad428be3f2 128 MiB Mon Dec 4 14:28:45 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:151 copied) 2023-12-04T14:36:05.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_18 info 2023-12-04T14:36:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_18 info' 2023-12-04T14:36:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_18 2023-12-04T14:36:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_18': 2023-12-04T14:36:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d391b309d7 2023-12-04T14:36:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d391b309d7 2023-12-04T14:36:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:35 2023 2023-12-04T14:36:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:28:55 2023 2023-12-04T14:36:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:35 2023 2023-12-04T14:36:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 9d431cf5-1ffc-4498-b3e0-5bba35366d30 2023-12-04T14:36:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:05.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.950 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_18 journal status 2023-12-04T14:36:05.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_18 journal status' 2023-12-04T14:36:05.951 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_18 2023-12-04T14:36:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_18 2023-12-04T14:36:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_18 snapshots 2023-12-04T14:36:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_18 snapshots' 2023-12-04T14:36:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_18 2023-12-04T14:36:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 169 snap 128 MiB Mon Dec 4 14:28:52 2023 user 2023-12-04T14:36:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 170 .mirror.non_primary.9d431cf5-1ffc-4498-b3e0-5bba35366d30.5550f487-2750-42b4-a6c0-9f2f8f9e183f 128 MiB Mon Dec 4 14:28:53 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:153 copied) 2023-12-04T14:36:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_19 info 2023-12-04T14:36:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_19 info' 2023-12-04T14:36:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_19 2023-12-04T14:36:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_19': 2023-12-04T14:36:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3248df615 2023-12-04T14:36:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3248df615 2023-12-04T14:36:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:48 2023 2023-12-04T14:36:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:29:02 2023 2023-12-04T14:36:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:48 2023 2023-12-04T14:36:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 44e764dc-83c2-44b2-bca6-672e14a04ca5 2023-12-04T14:36:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_19 journal status 2023-12-04T14:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_19 journal status' 2023-12-04T14:36:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_19 2023-12-04T14:36:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_19 2023-12-04T14:36:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_19 snapshots 2023-12-04T14:36:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_19 snapshots' 2023-12-04T14:36:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_19 2023-12-04T14:36:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 172 snap 128 MiB Mon Dec 4 14:28:59 2023 user 2023-12-04T14:36:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 173 .mirror.non_primary.44e764dc-83c2-44b2-bca6-672e14a04ca5.37e20a7a-4fe6-414d-8dad-0a1ba04fe76f 128 MiB Mon Dec 4 14:29:00 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:155 copied) 2023-12-04T14:36:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_2 info' 2023-12-04T14:36:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_2 info 2023-12-04T14:36:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_2 2023-12-04T14:36:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_2': 2023-12-04T14:36:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 11a2e0a5aaf0 2023-12-04T14:36:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.11a2e0a5aaf0 2023-12-04T14:36:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:17:42 2023 2023-12-04T14:36:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:27:06 2023 2023-12-04T14:36:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:17:42 2023 2023-12-04T14:36:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 9fe30a7e-3975-4be8-9ddc-ffdbf8f787e1 2023-12-04T14:36:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_2 journal status 2023-12-04T14:36:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_2 journal status' 2023-12-04T14:36:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_2 2023-12-04T14:36:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_2 2023-12-04T14:36:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_2 snapshots 2023-12-04T14:36:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_2 snapshots' 2023-12-04T14:36:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_2 2023-12-04T14:36:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 121 snap 128 MiB Mon Dec 4 14:27:03 2023 user 2023-12-04T14:36:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 122 .mirror.non_primary.9fe30a7e-3975-4be8-9ddc-ffdbf8f787e1.b97f5d70-b6af-446b-9420-4ce54a7f9f5b 128 MiB Mon Dec 4 14:27:04 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:121 copied) 2023-12-04T14:36:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_20 info 2023-12-04T14:36:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_20 info' 2023-12-04T14:36:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_20 2023-12-04T14:36:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_20': 2023-12-04T14:36:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3aff73751 2023-12-04T14:36:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3aff73751 2023-12-04T14:36:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:20:56 2023 2023-12-04T14:36:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:29:10 2023 2023-12-04T14:36:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:20:56 2023 2023-12-04T14:36:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: e7fd68b0-bd04-4c3c-af90-fd5517fef987 2023-12-04T14:36:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_20 journal status 2023-12-04T14:36:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_20 journal status' 2023-12-04T14:36:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_20 2023-12-04T14:36:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_20 2023-12-04T14:36:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_20 snapshots 2023-12-04T14:36:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_20 snapshots' 2023-12-04T14:36:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_20 2023-12-04T14:36:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 175 snap 128 MiB Mon Dec 4 14:29:07 2023 user 2023-12-04T14:36:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 176 .mirror.non_primary.e7fd68b0-bd04-4c3c-af90-fd5517fef987.c795b801-4980-48c3-8858-a76172fb43b6 128 MiB Mon Dec 4 14:29:08 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:157 copied) 2023-12-04T14:36:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_21 info' 2023-12-04T14:36:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_21 info 2023-12-04T14:36:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_21 2023-12-04T14:36:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_21': 2023-12-04T14:36:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d32911f4f7 2023-12-04T14:36:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d32911f4f7 2023-12-04T14:36:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:03 2023 2023-12-04T14:36:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:29:17 2023 2023-12-04T14:36:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:03 2023 2023-12-04T14:36:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 2a201e15-51f0-492a-8b9c-28ef7a490beb 2023-12-04T14:36:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_21 journal status 2023-12-04T14:36:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_21 journal status' 2023-12-04T14:36:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_21 2023-12-04T14:36:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_21 2023-12-04T14:36:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_21 snapshots 2023-12-04T14:36:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_21 snapshots' 2023-12-04T14:36:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_21 2023-12-04T14:36:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 178 snap 128 MiB Mon Dec 4 14:29:14 2023 user 2023-12-04T14:36:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 179 .mirror.non_primary.2a201e15-51f0-492a-8b9c-28ef7a490beb.f592996c-2584-4682-8af1-8961c0666f22 128 MiB Mon Dec 4 14:29:15 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:159 copied) 2023-12-04T14:36:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_22 info 2023-12-04T14:36:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_22 info' 2023-12-04T14:36:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_22 2023-12-04T14:36:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_22': 2023-12-04T14:36:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3318af3b2 2023-12-04T14:36:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3318af3b2 2023-12-04T14:36:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:13 2023 2023-12-04T14:36:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:29:24 2023 2023-12-04T14:36:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:13 2023 2023-12-04T14:36:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 059e9df9-d50d-4eb9-8306-2dcc0c369e9c 2023-12-04T14:36:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_22 journal status 2023-12-04T14:36:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_22 journal status' 2023-12-04T14:36:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_22 2023-12-04T14:36:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_22 2023-12-04T14:36:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_22 snapshots 2023-12-04T14:36:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_22 snapshots' 2023-12-04T14:36:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_22 2023-12-04T14:36:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 181 snap 128 MiB Mon Dec 4 14:29:21 2023 user 2023-12-04T14:36:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 182 .mirror.non_primary.059e9df9-d50d-4eb9-8306-2dcc0c369e9c.b2c1a163-3c8c-4eb8-9088-2581144a56e2 128 MiB Mon Dec 4 14:29:22 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:161 copied) 2023-12-04T14:36:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_23 info 2023-12-04T14:36:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_23 info' 2023-12-04T14:36:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_23 2023-12-04T14:36:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_23': 2023-12-04T14:36:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d326082b7b 2023-12-04T14:36:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d326082b7b 2023-12-04T14:36:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:26 2023 2023-12-04T14:36:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:29:30 2023 2023-12-04T14:36:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:26 2023 2023-12-04T14:36:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: e304857c-103b-447a-ae76-3be95494c2ac 2023-12-04T14:36:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_23 journal status 2023-12-04T14:36:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_23 journal status' 2023-12-04T14:36:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_23 2023-12-04T14:36:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_23 2023-12-04T14:36:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_23 snapshots 2023-12-04T14:36:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_23 snapshots' 2023-12-04T14:36:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_23 2023-12-04T14:36:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 184 snap 128 MiB Mon Dec 4 14:29:28 2023 user 2023-12-04T14:36:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 185 .mirror.non_primary.e304857c-103b-447a-ae76-3be95494c2ac.ba0a7cc9-24c6-466e-915b-70151acdef9e 128 MiB Mon Dec 4 14:29:29 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:163 copied) 2023-12-04T14:36:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_24 info 2023-12-04T14:36:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_24 info' 2023-12-04T14:36:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_24 2023-12-04T14:36:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_24': 2023-12-04T14:36:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d325210943 2023-12-04T14:36:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d325210943 2023-12-04T14:36:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:40 2023 2023-12-04T14:36:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:29:38 2023 2023-12-04T14:36:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:40 2023 2023-12-04T14:36:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: eb8df2f9-471f-4269-a2e3-70777410572c 2023-12-04T14:36:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:06.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_24 journal status 2023-12-04T14:36:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_24 journal status' 2023-12-04T14:36:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_24 2023-12-04T14:36:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_24 2023-12-04T14:36:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_24 snapshots 2023-12-04T14:36:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_24 snapshots' 2023-12-04T14:36:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_24 2023-12-04T14:36:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 187 snap 128 MiB Mon Dec 4 14:29:35 2023 user 2023-12-04T14:36:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 188 .mirror.non_primary.eb8df2f9-471f-4269-a2e3-70777410572c.43975dbe-2340-4a8c-b5ef-140a5c522b4e 128 MiB Mon Dec 4 14:29:36 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:165 copied) 2023-12-04T14:36:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_25 info' 2023-12-04T14:36:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_25 info 2023-12-04T14:36:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_25 2023-12-04T14:36:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_25': 2023-12-04T14:36:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3734b7b9f 2023-12-04T14:36:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3734b7b9f 2023-12-04T14:36:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:21:54 2023 2023-12-04T14:36:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:29:44 2023 2023-12-04T14:36:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:21:54 2023 2023-12-04T14:36:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 44871cb2-b60d-4e0e-8e66-9d9bdcbae254 2023-12-04T14:36:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_25 journal status 2023-12-04T14:36:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_25 journal status' 2023-12-04T14:36:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_25 2023-12-04T14:36:07.041 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_25 2023-12-04T14:36:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_25 snapshots 2023-12-04T14:36:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_25 snapshots' 2023-12-04T14:36:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_25 2023-12-04T14:36:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 190 snap 128 MiB Mon Dec 4 14:29:42 2023 user 2023-12-04T14:36:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 191 .mirror.non_primary.44871cb2-b60d-4e0e-8e66-9d9bdcbae254.4eeeac57-3020-4de9-8e22-d93e3db7c856 128 MiB Mon Dec 4 14:29:43 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:167 copied) 2023-12-04T14:36:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_26 info 2023-12-04T14:36:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_26 info' 2023-12-04T14:36:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_26 2023-12-04T14:36:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_26': 2023-12-04T14:36:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d389de6656 2023-12-04T14:36:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d389de6656 2023-12-04T14:36:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:07 2023 2023-12-04T14:36:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:29:50 2023 2023-12-04T14:36:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:07 2023 2023-12-04T14:36:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: edd4af1f-3c2b-4169-adc5-5d59c2cc5a61 2023-12-04T14:36:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_26 journal status 2023-12-04T14:36:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_26 journal status' 2023-12-04T14:36:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_26 2023-12-04T14:36:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_26 2023-12-04T14:36:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_26 snapshots 2023-12-04T14:36:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_26 snapshots' 2023-12-04T14:36:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_26 2023-12-04T14:36:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 193 snap 128 MiB Mon Dec 4 14:29:48 2023 user 2023-12-04T14:36:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 194 .mirror.non_primary.edd4af1f-3c2b-4169-adc5-5d59c2cc5a61.be89948d-a830-4fb1-a9c6-4e1bac5bed61 128 MiB Mon Dec 4 14:29:49 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:169 copied) 2023-12-04T14:36:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_27 info 2023-12-04T14:36:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_27 info' 2023-12-04T14:36:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_27 2023-12-04T14:36:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_27': 2023-12-04T14:36:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3fe9fbf08 2023-12-04T14:36:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3fe9fbf08 2023-12-04T14:36:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:13 2023 2023-12-04T14:36:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:29:57 2023 2023-12-04T14:36:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:13 2023 2023-12-04T14:36:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: e393e7c2-3dbe-4e5b-9517-da1c79d5fe47 2023-12-04T14:36:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_27 journal status' 2023-12-04T14:36:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_27 journal status 2023-12-04T14:36:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_27 2023-12-04T14:36:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_27 2023-12-04T14:36:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_27 snapshots 2023-12-04T14:36:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_27 snapshots' 2023-12-04T14:36:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_27 2023-12-04T14:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 196 snap 128 MiB Mon Dec 4 14:29:54 2023 user 2023-12-04T14:36:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 197 .mirror.non_primary.e393e7c2-3dbe-4e5b-9517-da1c79d5fe47.27111344-7b0f-42b2-92fe-f02ee6d12990 128 MiB Mon Dec 4 14:29:55 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:171 copied) 2023-12-04T14:36:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_28 info' 2023-12-04T14:36:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_28 info 2023-12-04T14:36:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_28 2023-12-04T14:36:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_28': 2023-12-04T14:36:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3a7b7e84b 2023-12-04T14:36:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3a7b7e84b 2023-12-04T14:36:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:25 2023 2023-12-04T14:36:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:30:03 2023 2023-12-04T14:36:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:25 2023 2023-12-04T14:36:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: ad2b0660-9c28-48f0-8b26-08f4860ad09e 2023-12-04T14:36:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_28 journal status 2023-12-04T14:36:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_28 journal status' 2023-12-04T14:36:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_28 2023-12-04T14:36:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_28 2023-12-04T14:36:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_28 snapshots 2023-12-04T14:36:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_28 snapshots' 2023-12-04T14:36:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_28 2023-12-04T14:36:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 199 snap 128 MiB Mon Dec 4 14:30:00 2023 user 2023-12-04T14:36:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 200 .mirror.non_primary.ad2b0660-9c28-48f0-8b26-08f4860ad09e.4bf44972-fa7e-4935-86c0-bcea359cbf44 128 MiB Mon Dec 4 14:30:01 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:173 copied) 2023-12-04T14:36:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_29 info' 2023-12-04T14:36:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_29 info 2023-12-04T14:36:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_29 2023-12-04T14:36:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_29': 2023-12-04T14:36:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d39517f19e 2023-12-04T14:36:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d39517f19e 2023-12-04T14:36:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:39 2023 2023-12-04T14:36:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:30:09 2023 2023-12-04T14:36:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:39 2023 2023-12-04T14:36:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 040b1033-d414-4ab1-bb84-83c330f7ff65 2023-12-04T14:36:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_29 journal status 2023-12-04T14:36:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_29 journal status' 2023-12-04T14:36:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_29 2023-12-04T14:36:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_29 2023-12-04T14:36:07.566 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_29 snapshots 2023-12-04T14:36:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_29 snapshots' 2023-12-04T14:36:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_29 2023-12-04T14:36:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 202 snap 128 MiB Mon Dec 4 14:30:06 2023 user 2023-12-04T14:36:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 203 .mirror.non_primary.040b1033-d414-4ab1-bb84-83c330f7ff65.4a6bd89d-d940-4151-a78c-b3441a08b4b4 128 MiB Mon Dec 4 14:30:07 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:175 copied) 2023-12-04T14:36:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_3 info 2023-12-04T14:36:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_3 info' 2023-12-04T14:36:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_3 2023-12-04T14:36:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_3': 2023-12-04T14:36:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3a1951f0 2023-12-04T14:36:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3a1951f0 2023-12-04T14:36:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:17:54 2023 2023-12-04T14:36:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:27:13 2023 2023-12-04T14:36:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:17:54 2023 2023-12-04T14:36:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: d5429052-81ab-46d6-ab3d-16695051435d 2023-12-04T14:36:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_3 journal status 2023-12-04T14:36:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_3 journal status' 2023-12-04T14:36:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_3 2023-12-04T14:36:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_3 2023-12-04T14:36:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_3 snapshots 2023-12-04T14:36:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_3 snapshots' 2023-12-04T14:36:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_3 2023-12-04T14:36:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 124 snap 128 MiB Mon Dec 4 14:27:10 2023 user 2023-12-04T14:36:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 125 .mirror.non_primary.d5429052-81ab-46d6-ab3d-16695051435d.5173ca71-2ebb-47d3-a5b1-35c0925be733 128 MiB Mon Dec 4 14:27:11 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:123 copied) 2023-12-04T14:36:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_30 info 2023-12-04T14:36:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_30 info' 2023-12-04T14:36:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_30 2023-12-04T14:36:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_30': 2023-12-04T14:36:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3ffbd15f1 2023-12-04T14:36:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3ffbd15f1 2023-12-04T14:36:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:46 2023 2023-12-04T14:36:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:30:17 2023 2023-12-04T14:36:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:46 2023 2023-12-04T14:36:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 233226c3-03eb-48bb-bc7e-fe8f5e6be5a9 2023-12-04T14:36:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_30 journal status 2023-12-04T14:36:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_30 journal status' 2023-12-04T14:36:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_30 2023-12-04T14:36:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_30 2023-12-04T14:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_30 snapshots 2023-12-04T14:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_30 snapshots' 2023-12-04T14:36:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_30 2023-12-04T14:36:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 205 snap 128 MiB Mon Dec 4 14:30:13 2023 user 2023-12-04T14:36:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 206 .mirror.non_primary.233226c3-03eb-48bb-bc7e-fe8f5e6be5a9.28a876f6-ad35-4ff8-b705-47a954076eee 128 MiB Mon Dec 4 14:30:14 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:177 copied) 2023-12-04T14:36:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_31 info 2023-12-04T14:36:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_31 info' 2023-12-04T14:36:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_31 2023-12-04T14:36:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_31': 2023-12-04T14:36:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3518272ec 2023-12-04T14:36:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3518272ec 2023-12-04T14:36:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:22:55 2023 2023-12-04T14:36:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:30:24 2023 2023-12-04T14:36:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:22:55 2023 2023-12-04T14:36:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 55aa29aa-3f60-443f-a765-b4d1a2257a41 2023-12-04T14:36:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_31 journal status 2023-12-04T14:36:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_31 journal status' 2023-12-04T14:36:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_31 2023-12-04T14:36:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_31 2023-12-04T14:36:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_31 snapshots 2023-12-04T14:36:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_31 snapshots' 2023-12-04T14:36:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_31 2023-12-04T14:36:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 208 snap 128 MiB Mon Dec 4 14:30:21 2023 user 2023-12-04T14:36:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 209 .mirror.non_primary.55aa29aa-3f60-443f-a765-b4d1a2257a41.ca22725a-e7b8-44cb-bcc3-1944b802e781 128 MiB Mon Dec 4 14:30:22 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:179 copied) 2023-12-04T14:36:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_32 info' 2023-12-04T14:36:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_32 info 2023-12-04T14:36:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_32 2023-12-04T14:36:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_32': 2023-12-04T14:36:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d36e345418 2023-12-04T14:36:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d36e345418 2023-12-04T14:36:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:01 2023 2023-12-04T14:36:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:30:30 2023 2023-12-04T14:36:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:01 2023 2023-12-04T14:36:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: da0438b0-0666-4c5d-bca2-497b527c3c8a 2023-12-04T14:36:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_32 journal status 2023-12-04T14:36:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_32 journal status' 2023-12-04T14:36:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_32 2023-12-04T14:36:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_32 2023-12-04T14:36:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_32 snapshots 2023-12-04T14:36:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_32 snapshots' 2023-12-04T14:36:08.096 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_32 2023-12-04T14:36:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 211 snap 128 MiB Mon Dec 4 14:30:28 2023 user 2023-12-04T14:36:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 212 .mirror.non_primary.da0438b0-0666-4c5d-bca2-497b527c3c8a.2e87f9b0-4354-4462-bc37-94430daf9371 128 MiB Mon Dec 4 14:30:29 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:181 copied) 2023-12-04T14:36:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_33 info' 2023-12-04T14:36:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_33 info 2023-12-04T14:36:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_33 2023-12-04T14:36:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_33': 2023-12-04T14:36:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3bd65a8c0 2023-12-04T14:36:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3bd65a8c0 2023-12-04T14:36:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:11 2023 2023-12-04T14:36:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:30:38 2023 2023-12-04T14:36:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:11 2023 2023-12-04T14:36:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 709187ee-c3f5-47e9-a97c-a08560690fe3 2023-12-04T14:36:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_33 journal status 2023-12-04T14:36:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_33 journal status' 2023-12-04T14:36:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_33 2023-12-04T14:36:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_33 2023-12-04T14:36:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_33 snapshots 2023-12-04T14:36:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_33 snapshots' 2023-12-04T14:36:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_33 2023-12-04T14:36:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 214 snap 128 MiB Mon Dec 4 14:30:35 2023 user 2023-12-04T14:36:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 215 .mirror.non_primary.709187ee-c3f5-47e9-a97c-a08560690fe3.beba3d41-297a-466e-8dbc-1c6f9ebfcd81 128 MiB Mon Dec 4 14:30:36 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:183 copied) 2023-12-04T14:36:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_34 info' 2023-12-04T14:36:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_34 info 2023-12-04T14:36:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_34 2023-12-04T14:36:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_34': 2023-12-04T14:36:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d32bdab1ea 2023-12-04T14:36:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d32bdab1ea 2023-12-04T14:36:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:25 2023 2023-12-04T14:36:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:30:44 2023 2023-12-04T14:36:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:25 2023 2023-12-04T14:36:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 53abed9b-8d5d-4a15-a122-712b8dc9c3d5 2023-12-04T14:36:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_34 journal status 2023-12-04T14:36:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_34 journal status' 2023-12-04T14:36:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_34 2023-12-04T14:36:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_34 2023-12-04T14:36:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_34 snapshots 2023-12-04T14:36:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_34 snapshots' 2023-12-04T14:36:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_34 2023-12-04T14:36:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 217 snap 128 MiB Mon Dec 4 14:30:42 2023 user 2023-12-04T14:36:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 218 .mirror.non_primary.53abed9b-8d5d-4a15-a122-712b8dc9c3d5.7c9f3e98-fff1-4238-9266-cb17eb0ab39f 128 MiB Mon Dec 4 14:30:43 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:185 copied) 2023-12-04T14:36:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_35 info' 2023-12-04T14:36:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_35 info 2023-12-04T14:36:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_35 2023-12-04T14:36:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_35': 2023-12-04T14:36:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d38f87952b 2023-12-04T14:36:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d38f87952b 2023-12-04T14:36:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:37 2023 2023-12-04T14:36:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:30:50 2023 2023-12-04T14:36:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:37 2023 2023-12-04T14:36:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 75dc1148-0c6b-4bf4-962c-9f4dc15daa97 2023-12-04T14:36:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_35 journal status 2023-12-04T14:36:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_35 journal status' 2023-12-04T14:36:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_35 2023-12-04T14:36:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_35 2023-12-04T14:36:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_35 snapshots 2023-12-04T14:36:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_35 snapshots' 2023-12-04T14:36:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_35 2023-12-04T14:36:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 220 snap 128 MiB Mon Dec 4 14:30:48 2023 user 2023-12-04T14:36:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 221 .mirror.non_primary.75dc1148-0c6b-4bf4-962c-9f4dc15daa97.62e433ba-0a04-4b66-9d3a-49f7ed0e45dc 128 MiB Mon Dec 4 14:30:49 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:187 copied) 2023-12-04T14:36:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_36 info 2023-12-04T14:36:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_36 info' 2023-12-04T14:36:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_36 2023-12-04T14:36:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_36': 2023-12-04T14:36:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3b8f9e819 2023-12-04T14:36:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3b8f9e819 2023-12-04T14:36:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:45 2023 2023-12-04T14:36:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:30:57 2023 2023-12-04T14:36:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:45 2023 2023-12-04T14:36:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: c88f1ade-8455-43f9-be6e-45b69b6f15a1 2023-12-04T14:36:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_36 journal status 2023-12-04T14:36:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_36 journal status' 2023-12-04T14:36:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_36 2023-12-04T14:36:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_36 2023-12-04T14:36:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_36 snapshots 2023-12-04T14:36:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_36 snapshots' 2023-12-04T14:36:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_36 2023-12-04T14:36:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 223 snap 128 MiB Mon Dec 4 14:30:54 2023 user 2023-12-04T14:36:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 224 .mirror.non_primary.c88f1ade-8455-43f9-be6e-45b69b6f15a1.41710bf7-f5a8-4963-8e69-bcf3a6bf3164 128 MiB Mon Dec 4 14:30:55 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:189 copied) 2023-12-04T14:36:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_37 info 2023-12-04T14:36:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_37 info' 2023-12-04T14:36:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_37 2023-12-04T14:36:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_37': 2023-12-04T14:36:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3bfecd311 2023-12-04T14:36:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3bfecd311 2023-12-04T14:36:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:23:57 2023 2023-12-04T14:36:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:23:57 2023 2023-12-04T14:36:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:23:57 2023 2023-12-04T14:36:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 573c9935-53e1-4562-b5e3-42d4b91a1ca2 2023-12-04T14:36:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_37 journal status 2023-12-04T14:36:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_37 journal status' 2023-12-04T14:36:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_37 2023-12-04T14:36:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_37 2023-12-04T14:36:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_37 snapshots 2023-12-04T14:36:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_37 snapshots' 2023-12-04T14:36:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_37 2023-12-04T14:36:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 226 snap 128 MiB Mon Dec 4 14:31:00 2023 user 2023-12-04T14:36:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 227 .mirror.non_primary.573c9935-53e1-4562-b5e3-42d4b91a1ca2.ad22db62-f8b9-483e-a654-039b310cefdc 128 MiB Mon Dec 4 14:31:01 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:191 copied) 2023-12-04T14:36:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_38 info 2023-12-04T14:36:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_38 info' 2023-12-04T14:36:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_38 2023-12-04T14:36:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_38': 2023-12-04T14:36:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3346a8c92 2023-12-04T14:36:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3346a8c92 2023-12-04T14:36:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:24:06 2023 2023-12-04T14:36:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:24:06 2023 2023-12-04T14:36:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:24:06 2023 2023-12-04T14:36:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: f19d0e0b-d80d-47ea-9448-f5c84f9d4cc6 2023-12-04T14:36:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_38 journal status 2023-12-04T14:36:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_38 journal status' 2023-12-04T14:36:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_38 2023-12-04T14:36:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_38 2023-12-04T14:36:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_38 snapshots 2023-12-04T14:36:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_38 snapshots' 2023-12-04T14:36:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_38 2023-12-04T14:36:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 105 .mirror.non_primary.f19d0e0b-d80d-47ea-9448-f5c84f9d4cc6.4958bafe-f4f8-4946-b6a0-cec58ba1fc0f 128 MiB Mon Dec 4 14:24:07 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:105 copied) 2023-12-04T14:36:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_39 info 2023-12-04T14:36:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_39 info' 2023-12-04T14:36:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_39 2023-12-04T14:36:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_39': 2023-12-04T14:36:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d36fd597d9 2023-12-04T14:36:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d36fd597d9 2023-12-04T14:36:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:24:14 2023 2023-12-04T14:36:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:24:14 2023 2023-12-04T14:36:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:24:14 2023 2023-12-04T14:36:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 622dfb37-1a24-4c28-8ac0-9f49b29eeead 2023-12-04T14:36:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_39 journal status 2023-12-04T14:36:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_39 journal status' 2023-12-04T14:36:08.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_39 2023-12-04T14:36:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_39 2023-12-04T14:36:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_39 snapshots 2023-12-04T14:36:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_39 snapshots' 2023-12-04T14:36:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_39 2023-12-04T14:36:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 106 .mirror.non_primary.622dfb37-1a24-4c28-8ac0-9f49b29eeead.3006911a-69ac-47be-80ca-38b57504a251 128 MiB Mon Dec 4 14:24:15 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:106 copied) 2023-12-04T14:36:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_4 info 2023-12-04T14:36:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_4 info' 2023-12-04T14:36:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_4 2023-12-04T14:36:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_4': 2023-12-04T14:36:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3c3da3ce1 2023-12-04T14:36:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3c3da3ce1 2023-12-04T14:36:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:06 2023 2023-12-04T14:36:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:27:19 2023 2023-12-04T14:36:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:06 2023 2023-12-04T14:36:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 60498ab7-7310-4365-aa39-30c7019acf9f 2023-12-04T14:36:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_4 journal status 2023-12-04T14:36:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_4 journal status' 2023-12-04T14:36:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_4 2023-12-04T14:36:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_4 2023-12-04T14:36:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_4 snapshots 2023-12-04T14:36:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_4 snapshots' 2023-12-04T14:36:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_4 2023-12-04T14:36:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 127 snap 128 MiB Mon Dec 4 14:27:17 2023 user 2023-12-04T14:36:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 128 .mirror.non_primary.60498ab7-7310-4365-aa39-30c7019acf9f.2dbfcf55-bf54-4320-b500-37378cfcebce 128 MiB Mon Dec 4 14:27:18 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:125 copied) 2023-12-04T14:36:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_40 info 2023-12-04T14:36:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_40 info' 2023-12-04T14:36:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_40 2023-12-04T14:36:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_40': 2023-12-04T14:36:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d387b8be81 2023-12-04T14:36:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d387b8be81 2023-12-04T14:36:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:24:29 2023 2023-12-04T14:36:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:24:29 2023 2023-12-04T14:36:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:24:29 2023 2023-12-04T14:36:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: f1ae4816-50e9-48b7-a351-3bb0cf0cd5bb 2023-12-04T14:36:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:09.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.234 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_40 journal status 2023-12-04T14:36:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_40 journal status' 2023-12-04T14:36:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_40 2023-12-04T14:36:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_40 2023-12-04T14:36:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_40 snapshots 2023-12-04T14:36:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_40 snapshots' 2023-12-04T14:36:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_40 2023-12-04T14:36:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 107 .mirror.non_primary.f1ae4816-50e9-48b7-a351-3bb0cf0cd5bb.6dadf350-96ff-43c1-98b8-b0dacdd4f0aa 128 MiB Mon Dec 4 14:24:30 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:107 copied) 2023-12-04T14:36:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_41 info 2023-12-04T14:36:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_41 info' 2023-12-04T14:36:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_41 2023-12-04T14:36:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_41': 2023-12-04T14:36:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d364b301d5 2023-12-04T14:36:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d364b301d5 2023-12-04T14:36:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:24:36 2023 2023-12-04T14:36:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:24:36 2023 2023-12-04T14:36:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:24:36 2023 2023-12-04T14:36:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 4985d9fc-9c9b-4cb0-a17e-3e72f3e74408 2023-12-04T14:36:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_41 journal status 2023-12-04T14:36:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_41 journal status' 2023-12-04T14:36:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_41 2023-12-04T14:36:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_41 2023-12-04T14:36:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_41 snapshots 2023-12-04T14:36:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_41 snapshots' 2023-12-04T14:36:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_41 2023-12-04T14:36:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 108 .mirror.non_primary.4985d9fc-9c9b-4cb0-a17e-3e72f3e74408.862b5ff3-8a2e-4198-9e4f-e9c7009f52d5 128 MiB Mon Dec 4 14:24:37 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:108 copied) 2023-12-04T14:36:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_42 info 2023-12-04T14:36:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_42 info' 2023-12-04T14:36:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_42 2023-12-04T14:36:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_42': 2023-12-04T14:36:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d34ff5991f 2023-12-04T14:36:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d34ff5991f 2023-12-04T14:36:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:24:44 2023 2023-12-04T14:36:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:24:44 2023 2023-12-04T14:36:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:24:44 2023 2023-12-04T14:36:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: fe0ec733-207f-4160-afad-a0d4633cee01 2023-12-04T14:36:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_42 journal status 2023-12-04T14:36:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_42 journal status' 2023-12-04T14:36:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_42 2023-12-04T14:36:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_42 2023-12-04T14:36:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_42 snapshots 2023-12-04T14:36:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_42 snapshots' 2023-12-04T14:36:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_42 2023-12-04T14:36:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 109 .mirror.non_primary.fe0ec733-207f-4160-afad-a0d4633cee01.8eb6665b-86cc-4dd1-9e92-db5659e43a53 128 MiB Mon Dec 4 14:24:45 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:109 copied) 2023-12-04T14:36:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_43 info 2023-12-04T14:36:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_43 info' 2023-12-04T14:36:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_43 2023-12-04T14:36:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_43': 2023-12-04T14:36:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d32ba45d48 2023-12-04T14:36:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d32ba45d48 2023-12-04T14:36:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:24:57 2023 2023-12-04T14:36:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:24:57 2023 2023-12-04T14:36:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:24:57 2023 2023-12-04T14:36:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 5a31e773-c4a1-4a79-9d13-74cc228e5bc2 2023-12-04T14:36:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_43 journal status 2023-12-04T14:36:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_43 journal status' 2023-12-04T14:36:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_43 2023-12-04T14:36:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_43 2023-12-04T14:36:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_43 snapshots 2023-12-04T14:36:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_43 snapshots' 2023-12-04T14:36:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_43 2023-12-04T14:36:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 110 .mirror.non_primary.5a31e773-c4a1-4a79-9d13-74cc228e5bc2.d48a3bb2-544b-471f-a5f6-b6a21bdd5d5c 128 MiB Mon Dec 4 14:24:57 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:110 copied) 2023-12-04T14:36:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_44 info 2023-12-04T14:36:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_44 info' 2023-12-04T14:36:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_44 2023-12-04T14:36:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_44': 2023-12-04T14:36:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d31873961e 2023-12-04T14:36:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d31873961e 2023-12-04T14:36:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:25:10 2023 2023-12-04T14:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:25:10 2023 2023-12-04T14:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:25:10 2023 2023-12-04T14:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 6705f07a-ea33-4f0a-b886-387067319889 2023-12-04T14:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_44 journal status 2023-12-04T14:36:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_44 journal status' 2023-12-04T14:36:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_44 2023-12-04T14:36:09.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_44 2023-12-04T14:36:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_44 snapshots 2023-12-04T14:36:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_44 snapshots' 2023-12-04T14:36:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_44 2023-12-04T14:36:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 111 .mirror.non_primary.6705f07a-ea33-4f0a-b886-387067319889.835df873-2e74-4271-b7fe-fba700431ffa 128 MiB Mon Dec 4 14:25:11 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:111 copied) 2023-12-04T14:36:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_45 info' 2023-12-04T14:36:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_45 info 2023-12-04T14:36:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_45 2023-12-04T14:36:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_45': 2023-12-04T14:36:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3f2a64cd4 2023-12-04T14:36:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3f2a64cd4 2023-12-04T14:36:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:25:18 2023 2023-12-04T14:36:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:25:18 2023 2023-12-04T14:36:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:25:18 2023 2023-12-04T14:36:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: bd2f1aeb-d6bf-4777-b4de-b7dbba39dc7b 2023-12-04T14:36:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_45 journal status 2023-12-04T14:36:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_45 journal status' 2023-12-04T14:36:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_45 2023-12-04T14:36:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_45 2023-12-04T14:36:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_45 snapshots 2023-12-04T14:36:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_45 snapshots' 2023-12-04T14:36:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_45 2023-12-04T14:36:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 112 .mirror.non_primary.bd2f1aeb-d6bf-4777-b4de-b7dbba39dc7b.90b198b1-c8ad-411f-8640-da1a0b14aeee 128 MiB Mon Dec 4 14:25:18 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:112 copied) 2023-12-04T14:36:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_46 info 2023-12-04T14:36:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_46 info' 2023-12-04T14:36:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_46 2023-12-04T14:36:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_46': 2023-12-04T14:36:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d378674d2b 2023-12-04T14:36:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d378674d2b 2023-12-04T14:36:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:25:25 2023 2023-12-04T14:36:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:25:25 2023 2023-12-04T14:36:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:25:25 2023 2023-12-04T14:36:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: f4eea075-92cf-457e-8d7a-77a594f0d72d 2023-12-04T14:36:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_46 journal status 2023-12-04T14:36:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_46 journal status' 2023-12-04T14:36:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_46 2023-12-04T14:36:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_46 2023-12-04T14:36:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_46 snapshots' 2023-12-04T14:36:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_46 snapshots 2023-12-04T14:36:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_46 2023-12-04T14:36:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 113 .mirror.non_primary.f4eea075-92cf-457e-8d7a-77a594f0d72d.ff5e2286-8e05-4d05-b9e3-99676206d49e 128 MiB Mon Dec 4 14:25:25 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:113 copied) 2023-12-04T14:36:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_47 info 2023-12-04T14:36:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_47 info' 2023-12-04T14:36:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_47 2023-12-04T14:36:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_47': 2023-12-04T14:36:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3c77cc0b 2023-12-04T14:36:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3c77cc0b 2023-12-04T14:36:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:25:40 2023 2023-12-04T14:36:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:25:40 2023 2023-12-04T14:36:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:25:40 2023 2023-12-04T14:36:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 1c9b05a7-06ae-420c-ab01-732da45bf605 2023-12-04T14:36:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_47 journal status 2023-12-04T14:36:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_47 journal status' 2023-12-04T14:36:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_47 2023-12-04T14:36:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_47 2023-12-04T14:36:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_47 snapshots 2023-12-04T14:36:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_47 snapshots' 2023-12-04T14:36:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_47 2023-12-04T14:36:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 114 .mirror.non_primary.1c9b05a7-06ae-420c-ab01-732da45bf605.377b9e26-420c-44fa-9377-490a13b58679 128 MiB Mon Dec 4 14:25:40 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:114 copied) 2023-12-04T14:36:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_48 info' 2023-12-04T14:36:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_48 info 2023-12-04T14:36:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_48 2023-12-04T14:36:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_48': 2023-12-04T14:36:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3b790c0a3 2023-12-04T14:36:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3b790c0a3 2023-12-04T14:36:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:25:55 2023 2023-12-04T14:36:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:25:55 2023 2023-12-04T14:36:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:25:55 2023 2023-12-04T14:36:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 75662a66-3ee6-422c-bc5b-2ed60a3a9402 2023-12-04T14:36:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_48 journal status 2023-12-04T14:36:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_48 journal status' 2023-12-04T14:36:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_48 2023-12-04T14:36:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_48 2023-12-04T14:36:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_48 snapshots 2023-12-04T14:36:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_48 snapshots' 2023-12-04T14:36:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_48 2023-12-04T14:36:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 115 .mirror.non_primary.75662a66-3ee6-422c-bc5b-2ed60a3a9402.788725f2-7602-4138-82d7-6c4e9e60e942 128 MiB Mon Dec 4 14:25:56 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:115 copied) 2023-12-04T14:36:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_49 info 2023-12-04T14:36:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_49 info' 2023-12-04T14:36:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_49 2023-12-04T14:36:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_49': 2023-12-04T14:36:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3d7ec3591 2023-12-04T14:36:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3d7ec3591 2023-12-04T14:36:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:26:05 2023 2023-12-04T14:36:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:26:05 2023 2023-12-04T14:36:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:26:05 2023 2023-12-04T14:36:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: c368a0ff-abe8-4e7e-bf74-d33daf9fed3d 2023-12-04T14:36:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_49 journal status 2023-12-04T14:36:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_49 journal status' 2023-12-04T14:36:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_49 2023-12-04T14:36:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_49 2023-12-04T14:36:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_49 snapshots 2023-12-04T14:36:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_49 snapshots' 2023-12-04T14:36:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_49 2023-12-04T14:36:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 116 .mirror.non_primary.c368a0ff-abe8-4e7e-bf74-d33daf9fed3d.8cd71080-576a-43da-b919-78c1d7dce50b 128 MiB Mon Dec 4 14:26:06 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:116 copied) 2023-12-04T14:36:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_5 info 2023-12-04T14:36:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_5 info' 2023-12-04T14:36:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_5 2023-12-04T14:36:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_5': 2023-12-04T14:36:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d33c6af833 2023-12-04T14:36:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d33c6af833 2023-12-04T14:36:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:17 2023 2023-12-04T14:36:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:27:27 2023 2023-12-04T14:36:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:17 2023 2023-12-04T14:36:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 90bc0c25-b4ac-4b4a-8f12-bbe6c286b2cd 2023-12-04T14:36:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_5 journal status' 2023-12-04T14:36:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_5 journal status 2023-12-04T14:36:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_5 2023-12-04T14:36:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_5 2023-12-04T14:36:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_5 snapshots 2023-12-04T14:36:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_5 snapshots' 2023-12-04T14:36:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_5 2023-12-04T14:36:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 130 snap 128 MiB Mon Dec 4 14:27:24 2023 user 2023-12-04T14:36:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 131 .mirror.non_primary.90bc0c25-b4ac-4b4a-8f12-bbe6c286b2cd.03cf1186-1203-4db3-89ab-801c5fdf74d2 128 MiB Mon Dec 4 14:27:25 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:127 copied) 2023-12-04T14:36:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_50 info 2023-12-04T14:36:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_50 info' 2023-12-04T14:36:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_50 2023-12-04T14:36:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_50': 2023-12-04T14:36:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 1 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d35fdc9a51 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d35fdc9a51 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:26:11 2023 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:26:11 2023 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:26:11 2023 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: b6c8de6f-5527-43d7-be14-65c94f60c736 2023-12-04T14:36:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_50 journal status 2023-12-04T14:36:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_50 journal status' 2023-12-04T14:36:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_50 2023-12-04T14:36:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_50 2023-12-04T14:36:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_50 snapshots 2023-12-04T14:36:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_50 snapshots' 2023-12-04T14:36:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_50 2023-12-04T14:36:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 117 .mirror.non_primary.b6c8de6f-5527-43d7-be14-65c94f60c736.89fc17bd-4cd2-46d6-bea6-e0d3b6ccfeaf 128 MiB Mon Dec 4 14:26:12 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:117 copied) 2023-12-04T14:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_6 info 2023-12-04T14:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_6 info' 2023-12-04T14:36:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_6 2023-12-04T14:36:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_6': 2023-12-04T14:36:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3d00f5c86 2023-12-04T14:36:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3d00f5c86 2023-12-04T14:36:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:26 2023 2023-12-04T14:36:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:27:34 2023 2023-12-04T14:36:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:26 2023 2023-12-04T14:36:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 481d8ed8-0c1d-4f88-8ce7-bbcfd2363766 2023-12-04T14:36:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_6 journal status 2023-12-04T14:36:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_6 journal status' 2023-12-04T14:36:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_6 2023-12-04T14:36:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_6 2023-12-04T14:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_6 snapshots 2023-12-04T14:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_6 snapshots' 2023-12-04T14:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_6 2023-12-04T14:36:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 133 snap 128 MiB Mon Dec 4 14:27:31 2023 user 2023-12-04T14:36:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 134 .mirror.non_primary.481d8ed8-0c1d-4f88-8ce7-bbcfd2363766.699f67ce-cb63-4cd9-8b2f-7203a14d3a4d 128 MiB Mon Dec 4 14:27:32 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:129 copied) 2023-12-04T14:36:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_7 info 2023-12-04T14:36:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_7 info' 2023-12-04T14:36:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_7 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_7': 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3d23474f 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3d23474f 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:36 2023 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:27:40 2023 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:36 2023 2023-12-04T14:36:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 431573ac-8014-4867-b7be-738ee54b8f15 2023-12-04T14:36:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_7 journal status 2023-12-04T14:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_7 journal status' 2023-12-04T14:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_7 2023-12-04T14:36:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_7 2023-12-04T14:36:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_7 snapshots 2023-12-04T14:36:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_7 snapshots' 2023-12-04T14:36:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_7 2023-12-04T14:36:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 136 snap 128 MiB Mon Dec 4 14:27:38 2023 user 2023-12-04T14:36:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 137 .mirror.non_primary.431573ac-8014-4867-b7be-738ee54b8f15.a34ca499-7cea-4a35-9823-17b6dfd78318 128 MiB Mon Dec 4 14:27:39 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:131 copied) 2023-12-04T14:36:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_8 info 2023-12-04T14:36:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_8 info' 2023-12-04T14:36:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_8 2023-12-04T14:36:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_8': 2023-12-04T14:36:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3ad137cc4 2023-12-04T14:36:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3ad137cc4 2023-12-04T14:36:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:42 2023 2023-12-04T14:36:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:27:48 2023 2023-12-04T14:36:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:42 2023 2023-12-04T14:36:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: 14168f82-d9d2-41fc-99ed-f1205d8ef156 2023-12-04T14:36:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_8 journal status 2023-12-04T14:36:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_8 journal status' 2023-12-04T14:36:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_8 2023-12-04T14:36:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_8 2023-12-04T14:36:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_8 snapshots 2023-12-04T14:36:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_8 snapshots' 2023-12-04T14:36:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_8 2023-12-04T14:36:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 139 snap 128 MiB Mon Dec 4 14:27:45 2023 user 2023-12-04T14:36:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 140 .mirror.non_primary.14168f82-d9d2-41fc-99ed-f1205d8ef156.3563cd88-4e14-4abc-b9d2-40d79d9a9b25 128 MiB Mon Dec 4 14:27:46 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:133 copied) 2023-12-04T14:36:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2023-12-04T14:36:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_9 info 2023-12-04T14:36:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_9 info' 2023-12-04T14:36:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_9 2023-12-04T14:36:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:rbd image 'image_9': 2023-12-04T14:36:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: size 128 MiB in 32 objects 2023-12-04T14:36:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: order 22 (4 MiB objects) 2023-12-04T14:36:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: snapshot_count: 2 2023-12-04T14:36:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: id: 10d3857e5ce3 2023-12-04T14:36:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: block_name_prefix: rbd_data.10d3857e5ce3 2023-12-04T14:36:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: format: 2 2023-12-04T14:36:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: features: layering, non-primary 2023-12-04T14:36:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: op_features: 2023-12-04T14:36:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: flags: 2023-12-04T14:36:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: create_timestamp: Mon Dec 4 14:18:52 2023 2023-12-04T14:36:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: access_timestamp: Mon Dec 4 14:27:54 2023 2023-12-04T14:36:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: modify_timestamp: Mon Dec 4 14:18:52 2023 2023-12-04T14:36:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring state: enabled 2023-12-04T14:36:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring mode: snapshot 2023-12-04T14:36:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring global id: fc8c6534-5c92-48ae-b43e-099c17bc6417 2023-12-04T14:36:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: mirroring primary: false 2023-12-04T14:36:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_9 journal status 2023-12-04T14:36:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_9 journal status' 2023-12-04T14:36:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_9 2023-12-04T14:36:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: journaling is not enabled for image image_9 2023-12-04T14:36:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image image_9 snapshots 2023-12-04T14:36:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'image image_9 snapshots' 2023-12-04T14:36:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_9 2023-12-04T14:36:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2023-12-04T14:36:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 142 snap 128 MiB Mon Dec 4 14:27:52 2023 user 2023-12-04T14:36:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 143 .mirror.non_primary.fc8c6534-5c92-48ae-b43e-099c17bc6417.d72bebe3-427f-4a2c-9948-a6dcd888bb47 128 MiB Mon Dec 4 14:27:53 2023 mirror (non-primary peer_uuids:[] f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba:135 copied) 2023-12-04T14:36:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror rbd_mirroring omap vals 2023-12-04T14:36:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror rbd_mirroring omap vals' 2023-12-04T14:36:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirroring 2023-12-04T14:36:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_040b1033-d414-4ab1-bb84-83c330f7ff65 2023-12-04T14:36:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 39 35 31 37 66 31 39 65 |....10d39517f19e| 2023-12-04T14:36:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_059e9df9-d50d-4eb9-8306-2dcc0c369e9c 2023-12-04T14:36:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 33 31 38 61 66 33 62 32 |....10d3318af3b2| 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_0d9ffbfe-5f0b-4261-82aa-d3c625dda130 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 66 64 38 39 65 65 65 31 |....10d3fd89eee1| 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_0e39536e-cf2c-4b79-ae6d-a5d1ede7920c 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 62 33 61 64 30 34 31 37 |....10d3b3ad0417| 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_14168f82-d9d2-41fc-99ed-f1205d8ef156 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 61 64 31 33 37 63 63 34 |....10d3ad137cc4| 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_1c9b05a7-06ae-420c-ab01-732da45bf605 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (15 bytes) : 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0b 00 00 00 31 30 64 33 63 37 37 63 63 30 62 |....10d3c77cc0b| 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000000f 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_233226c3-03eb-48bb-bc7e-fe8f5e6be5a9 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 66 66 62 64 31 35 66 31 |....10d3ffbd15f1| 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_2a201e15-51f0-492a-8b9c-28ef7a490beb 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 32 39 31 31 66 34 66 37 |....10d32911f4f7| 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_431573ac-8014-4867-b7be-738ee54b8f15 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (15 bytes) : 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0b 00 00 00 31 30 64 33 64 32 33 34 37 34 66 |....10d3d23474f| 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000000f 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_44871cb2-b60d-4e0e-8e66-9d9bdcbae254 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 37 33 34 62 37 62 39 66 |....10d3734b7b9f| 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_44e764dc-83c2-44b2-bca6-672e14a04ca5 2023-12-04T14:36:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 32 34 38 64 66 36 31 35 |....10d3248df615| 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_481d8ed8-0c1d-4f88-8ce7-bbcfd2363766 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 64 30 30 66 35 63 38 36 |....10d3d00f5c86| 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_4985d9fc-9c9b-4cb0-a17e-3e72f3e74408 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 36 34 62 33 30 31 64 35 |....10d364b301d5| 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_53abed9b-8d5d-4a15-a122-712b8dc9c3d5 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 32 62 64 61 62 31 65 61 |....10d32bdab1ea| 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_55aa29aa-3f60-443f-a765-b4d1a2257a41 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 35 31 38 32 37 32 65 63 |....10d3518272ec| 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_573c9935-53e1-4562-b5e3-42d4b91a1ca2 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 62 66 65 63 64 33 31 31 |....10d3bfecd311| 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_5854cc13-6872-45e6-8c82-645c028f7199 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 65 61 34 38 64 66 36 34 |....10d3ea48df64| 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_5a31e773-c4a1-4a79-9d13-74cc228e5bc2 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 32 62 61 34 35 64 34 38 |....10d32ba45d48| 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_5f322e9d-b400-4d44-81e0-ffdd216398b2 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 37 31 39 31 36 38 61 34 |....10d3719168a4| 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_60498ab7-7310-4365-aa39-30c7019acf9f 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 63 33 64 61 33 63 65 31 |....10d3c3da3ce1| 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_622dfb37-1a24-4c28-8ac0-9f49b29eeead 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 36 66 64 35 39 37 64 39 |....10d36fd597d9| 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_6705f07a-ea33-4f0a-b886-387067319889 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 31 38 37 33 39 36 31 65 |....10d31873961e| 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_709187ee-c3f5-47e9-a97c-a08560690fe3 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 62 64 36 35 61 38 63 30 |....10d3bd65a8c0| 2023-12-04T14:36:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_75662a66-3ee6-422c-bc5b-2ed60a3a9402 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 62 37 39 30 63 30 61 33 |....10d3b790c0a3| 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_75dc1148-0c6b-4bf4-962c-9f4dc15daa97 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 38 66 38 37 39 35 32 62 |....10d38f87952b| 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_7bfe12f9-a769-4f59-bca6-8c32a93f3809 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 66 30 35 65 38 37 61 61 |....10d3f05e87aa| 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_8794a2f7-791e-4d63-afdb-a1f297b316fa 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 66 34 36 34 61 62 30 35 |....10d3f464ab05| 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_90bc0c25-b4ac-4b4a-8f12-bbe6c286b2cd 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 33 63 36 61 66 38 33 33 |....10d33c6af833| 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_99047729-2de2-4c91-90dc-1e2bfe4f2b5f 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 32 62 30 34 63 36 62 64 |....10d32b04c6bd| 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_9d431cf5-1ffc-4498-b3e0-5bba35366d30 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 39 31 62 33 30 39 64 37 |....10d391b309d7| 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_9fe30a7e-3975-4be8-9ddc-ffdbf8f787e1 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 31 61 32 65 30 61 35 61 61 66 30 |....11a2e0a5aaf0| 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_ac478d96-184c-4eaa-8c62-f6f0e8288dff 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 34 31 37 34 38 33 32 61 |....10d34174832a| 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_ad2b0660-9c28-48f0-8b26-08f4860ad09e 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 61 37 62 37 65 38 34 62 |....10d3a7b7e84b| 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_b6c8de6f-5527-43d7-be14-65c94f60c736 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 35 66 64 63 39 61 35 31 |....10d35fdc9a51| 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_b928065e-ca9b-41a2-9e4f-db3d8351e631 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 31 61 32 34 36 66 34 35 35 34 33 |....11a246f45543| 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_bd2f1aeb-d6bf-4777-b4de-b7dbba39dc7b 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 66 32 61 36 34 63 64 34 |....10d3f2a64cd4| 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_c368a0ff-abe8-4e7e-bf74-d33daf9fed3d 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 64 37 65 63 33 35 39 31 |....10d3d7ec3591| 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_c88f1ade-8455-43f9-be6e-45b69b6f15a1 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 62 38 66 39 65 38 31 39 |....10d3b8f9e819| 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_d5429052-81ab-46d6-ab3d-16695051435d 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (15 bytes) : 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0b 00 00 00 31 30 64 33 61 31 39 35 31 66 30 |....10d3a1951f0| 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000000f 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_da0438b0-0666-4c5d-bca2-497b527c3c8a 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 36 65 33 34 35 34 31 38 |....10d36e345418| 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_e304857c-103b-447a-ae76-3be95494c2ac 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 32 36 30 38 32 62 37 62 |....10d326082b7b| 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_e393e7c2-3dbe-4e5b-9517-da1c79d5fe47 2023-12-04T14:36:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 66 65 39 66 62 66 30 38 |....10d3fe9fbf08| 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_e7fd68b0-bd04-4c3c-af90-fd5517fef987 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 61 66 66 37 33 37 35 31 |....10d3aff73751| 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_eb8df2f9-471f-4269-a2e3-70777410572c 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 32 35 32 31 30 39 34 33 |....10d325210943| 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_edd4af1f-3c2b-4169-adc5-5d59c2cc5a61 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 38 39 64 65 36 36 35 36 |....10d389de6656| 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_f19d0e0b-d80d-47ea-9448-f5c84f9d4cc6 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 33 34 36 61 38 63 39 32 |....10d3346a8c92| 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_f1ae4816-50e9-48b7-a351-3bb0cf0cd5bb 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 38 37 62 38 62 65 38 31 |....10d387b8be81| 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_f4eea075-92cf-457e-8d7a-77a594f0d72d 2023-12-04T14:36:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 37 38 36 37 34 64 32 62 |....10d378674d2b| 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_fc8c6534-5c92-48ae-b43e-099c17bc6417 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 38 35 37 65 35 63 65 33 |....10d3857e5ce3| 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:global_fe0ec733-207f-4160-afad-a0d4633cee01 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (16 bytes) : 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 0c 00 00 00 31 30 64 33 34 66 66 35 39 39 31 66 |....10d34ff5991f| 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d31873961e 2023-12-04T14:36:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 37 30 35 66 30 |..*...$...6705f0| 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 37 61 2d 65 61 33 33 2d 34 66 30 61 2d 62 38 38 |7a-ea33-4f0a-b88| 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 36 2d 33 38 37 30 36 37 33 31 39 38 38 39 01 01 |6-387067319889..| 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3248df615 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 34 65 37 36 34 |..*...$...44e764| 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 64 63 2d 38 33 63 32 2d 34 34 62 32 2d 62 63 61 |dc-83c2-44b2-bca| 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 36 2d 36 37 32 65 31 34 61 30 34 63 61 35 01 01 |6-672e14a04ca5..| 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d325210943 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 62 38 64 66 32 |..*...$...eb8df2| 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 66 39 2d 34 37 31 66 2d 34 32 36 39 2d 61 32 65 |f9-471f-4269-a2e| 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 33 2d 37 30 37 37 37 34 31 30 35 37 32 63 01 01 |3-70777410572c..| 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d326082b7b 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 33 30 34 38 35 |..*...$...e30485| 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 37 63 2d 31 30 33 62 2d 34 34 37 61 2d 61 65 37 |7c-103b-447a-ae7| 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 36 2d 33 62 65 39 35 34 39 34 63 32 61 63 01 01 |6-3be95494c2ac..| 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d32911f4f7 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 32 61 32 30 31 65 |..*...$...2a201e| 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 31 35 2d 35 31 66 30 2d 34 39 32 61 2d 38 62 39 |15-51f0-492a-8b9| 2023-12-04T14:36:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 63 2d 32 38 65 66 37 61 34 39 30 62 65 62 01 01 |c-28ef7a490beb..| 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d32b04c6bd 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 39 30 34 37 37 |..*...$...990477| 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 32 39 2d 32 64 65 32 2d 34 63 39 31 2d 39 30 64 |29-2de2-4c91-90d| 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 63 2d 31 65 32 62 66 65 34 66 32 62 35 66 01 01 |c-1e2bfe4f2b5f..| 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d32ba45d48 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 61 33 31 65 37 |..*...$...5a31e7| 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 37 33 2d 63 34 61 31 2d 34 61 37 39 2d 39 64 31 |73-c4a1-4a79-9d1| 2023-12-04T14:36:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 33 2d 37 34 63 63 32 32 38 65 35 62 63 32 01 01 |3-74cc228e5bc2..| 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d32bdab1ea 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 33 61 62 65 64 |..*...$...53abed| 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 39 62 2d 38 64 35 64 2d 34 61 31 35 2d 61 31 32 |9b-8d5d-4a15-a12| 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 32 2d 37 31 32 62 38 64 63 39 63 33 64 35 01 01 |2-712b8dc9c3d5..| 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3318af3b2 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 35 39 65 39 64 |..*...$...059e9d| 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 66 39 2d 64 35 30 64 2d 34 65 62 39 2d 38 33 30 |f9-d50d-4eb9-830| 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 36 2d 32 64 63 63 30 63 33 36 39 65 39 63 01 01 |6-2dcc0c369e9c..| 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3346a8c92 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 31 39 64 30 65 |..*...$...f19d0e| 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 30 62 2d 64 38 30 64 2d 34 37 65 61 2d 39 34 34 |0b-d80d-47ea-944| 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 38 2d 66 35 63 38 34 66 39 64 34 63 63 36 01 01 |8-f5c84f9d4cc6..| 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d33c6af833 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 30 62 63 30 63 |..*...$...90bc0c| 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 32 35 2d 62 34 61 63 2d 34 62 34 61 2d 38 66 31 |25-b4ac-4b4a-8f1| 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 32 2d 62 62 65 36 63 32 38 36 62 32 63 64 01 01 |2-bbe6c286b2cd..| 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d34174832a 2023-12-04T14:36:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 63 34 37 38 64 |..*...$...ac478d| 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 39 36 2d 31 38 34 63 2d 34 65 61 61 2d 38 63 36 |96-184c-4eaa-8c6| 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 32 2d 66 36 66 30 65 38 32 38 38 64 66 66 01 01 |2-f6f0e8288dff..| 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d34ff5991f 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 65 30 65 63 37 |..*...$...fe0ec7| 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 33 33 2d 32 30 37 66 2d 34 31 36 30 2d 61 66 61 |33-207f-4160-afa| 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 64 2d 61 30 64 34 36 33 33 63 65 65 30 31 01 01 |d-a0d4633cee01..| 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3518272ec 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 35 61 61 32 39 |..*...$...55aa29| 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 61 61 2d 33 66 36 30 2d 34 34 33 66 2d 61 37 36 |aa-3f60-443f-a76| 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 35 2d 62 34 64 31 61 32 32 35 37 61 34 31 01 01 |5-b4d1a2257a41..| 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d35fdc9a51 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 36 63 38 64 65 |..*...$...b6c8de| 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 36 66 2d 35 35 32 37 2d 34 33 64 37 2d 62 65 31 |6f-5527-43d7-be1| 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 34 2d 36 35 63 39 34 66 36 30 63 37 33 36 01 01 |4-65c94f60c736..| 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d364b301d5 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 39 38 35 64 39 |..*...$...4985d9| 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 66 63 2d 39 63 39 62 2d 34 63 62 30 2d 61 31 37 |fc-9c9b-4cb0-a17| 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 65 2d 33 65 37 32 66 33 65 37 34 34 30 38 01 01 |e-3e72f3e74408..| 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d36e345418 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 61 30 34 33 38 |..*...$...da0438| 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 62 30 2d 30 36 36 36 2d 34 63 35 64 2d 62 63 61 |b0-0666-4c5d-bca| 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 32 2d 34 39 37 62 35 32 37 63 33 63 38 61 01 01 |2-497b527c3c8a..| 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d36fd597d9 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 32 32 64 66 62 |..*...$...622dfb| 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 33 37 2d 31 61 32 34 2d 34 63 32 38 2d 38 61 63 |37-1a24-4c28-8ac| 2023-12-04T14:36:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 30 2d 39 66 34 39 62 32 39 65 65 65 61 64 01 01 |0-9f49b29eeead..| 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3719168a4 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 66 33 32 32 65 |..*...$...5f322e| 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 39 64 2d 62 34 30 30 2d 34 64 34 34 2d 38 31 65 |9d-b400-4d44-81e| 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 30 2d 66 66 64 64 32 31 36 33 39 38 62 32 01 01 |0-ffdd216398b2..| 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3734b7b9f 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 34 38 37 31 63 |..*...$...44871c| 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 62 32 2d 62 36 30 64 2d 34 65 30 65 2d 38 65 36 |b2-b60d-4e0e-8e6| 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 36 2d 39 64 39 62 64 63 62 61 65 32 35 34 01 01 |6-9d9bdcbae254..| 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d378674d2b 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 34 65 65 61 30 |..*...$...f4eea0| 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 37 35 2d 39 32 63 66 2d 34 35 37 65 2d 38 64 37 |75-92cf-457e-8d7| 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 61 2d 37 37 61 35 39 34 66 30 64 37 32 64 01 01 |a-77a594f0d72d..| 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3857e5ce3 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 63 38 63 36 35 |..*...$...fc8c65| 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 33 34 2d 35 63 39 32 2d 34 38 61 65 2d 62 34 33 |34-5c92-48ae-b43| 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 65 2d 30 39 39 63 31 37 62 63 36 34 31 37 01 01 |e-099c17bc6417..| 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d387b8be81 2023-12-04T14:36:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 31 61 65 34 38 |..*...$...f1ae48| 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 31 36 2d 35 30 65 39 2d 34 38 62 37 2d 61 33 35 |16-50e9-48b7-a35| 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 31 2d 33 62 62 30 63 66 30 63 64 35 62 62 01 01 |1-3bb0cf0cd5bb..| 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d389de6656 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 64 64 34 61 66 |..*...$...edd4af| 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 31 66 2d 33 63 32 62 2d 34 31 36 39 2d 61 64 63 |1f-3c2b-4169-adc| 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 35 2d 35 64 35 39 63 32 63 63 35 61 36 31 01 01 |5-5d59c2cc5a61..| 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d38f87952b 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 35 64 63 31 31 |..*...$...75dc11| 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 34 38 2d 30 63 36 62 2d 34 62 66 34 2d 39 36 32 |48-0c6b-4bf4-962| 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 63 2d 39 66 34 64 63 31 35 64 61 61 39 37 01 01 |c-9f4dc15daa97..| 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d391b309d7 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 64 34 33 31 63 |..*...$...9d431c| 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 66 35 2d 31 66 66 63 2d 34 34 39 38 2d 62 33 65 |f5-1ffc-4498-b3e| 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 30 2d 35 62 62 61 33 35 33 36 36 64 33 30 01 01 |0-5bba35366d30..| 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d39517f19e 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 34 30 62 31 30 |..*...$...040b10| 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 33 33 2d 64 34 31 34 2d 34 61 62 31 2d 62 62 38 |33-d414-4ab1-bb8| 2023-12-04T14:36:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 34 2d 38 33 63 33 33 30 66 37 66 66 36 35 01 01 |4-83c330f7ff65..| 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3a1951f0 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 35 34 32 39 30 |..*...$...d54290| 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 35 32 2d 38 31 61 62 2d 34 36 64 36 2d 61 62 33 |52-81ab-46d6-ab3| 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 64 2d 31 36 36 39 35 30 35 31 34 33 35 64 01 01 |d-16695051435d..| 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3a7b7e84b 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 64 32 62 30 36 |..*...$...ad2b06| 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 36 30 2d 39 63 32 38 2d 34 38 66 30 2d 38 62 32 |60-9c28-48f0-8b2| 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 36 2d 30 38 66 34 38 36 30 61 64 30 39 65 01 01 |6-08f4860ad09e..| 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3ad137cc4 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 34 31 36 38 66 |..*...$...14168f| 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 38 32 2d 64 39 64 32 2d 34 31 66 63 2d 39 39 65 |82-d9d2-41fc-99e| 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 64 2d 66 31 32 30 35 64 38 65 66 31 35 36 01 01 |d-f1205d8ef156..| 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3aff73751 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 37 66 64 36 38 |..*...$...e7fd68| 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 62 30 2d 62 64 30 34 2d 34 63 33 63 2d 61 66 39 |b0-bd04-4c3c-af9| 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 30 2d 66 64 35 35 31 37 66 65 66 39 38 37 01 01 |0-fd5517fef987..| 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3b3ad0417 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 65 33 39 35 33 |..*...$...0e3953| 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 36 65 2d 63 66 32 63 2d 34 62 37 39 2d 61 65 36 |6e-cf2c-4b79-ae6| 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 64 2d 61 35 64 31 65 64 65 37 39 32 30 63 01 01 |d-a5d1ede7920c..| 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3b790c0a3 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 35 36 36 32 61 |..*...$...75662a| 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 36 36 2d 33 65 65 36 2d 34 32 32 63 2d 62 63 35 |66-3ee6-422c-bc5| 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 62 2d 32 65 64 36 30 61 33 61 39 34 30 32 01 01 |b-2ed60a3a9402..| 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3b8f9e819 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 38 38 66 31 61 |..*...$...c88f1a| 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 64 65 2d 38 34 35 35 2d 34 33 66 39 2d 62 65 36 |de-8455-43f9-be6| 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 65 2d 34 35 62 36 39 62 36 66 31 35 61 31 01 01 |e-45b69b6f15a1..| 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3bd65a8c0 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 30 39 31 38 37 |..*...$...709187| 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 65 65 2d 63 33 66 35 2d 34 37 65 39 2d 61 39 37 |ee-c3f5-47e9-a97| 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 63 2d 61 30 38 35 36 30 36 39 30 66 65 33 01 01 |c-a08560690fe3..| 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3bfecd311 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 37 33 63 39 39 |..*...$...573c99| 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 33 35 2d 35 33 65 31 2d 34 35 36 32 2d 62 35 65 |35-53e1-4562-b5e| 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 33 2d 34 32 64 34 62 39 31 61 31 63 61 32 01 01 |3-42d4b91a1ca2..| 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3c3da3ce1 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 30 34 39 38 61 |..*...$...60498a| 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 62 37 2d 37 33 31 30 2d 34 33 36 35 2d 61 61 33 |b7-7310-4365-aa3| 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 39 2d 33 30 63 37 30 31 39 61 63 66 39 66 01 01 |9-30c7019acf9f..| 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3c77cc0b 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 63 39 62 30 35 |..*...$...1c9b05| 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 61 37 2d 30 36 61 65 2d 34 32 30 63 2d 61 62 30 |a7-06ae-420c-ab0| 2023-12-04T14:36:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 31 2d 37 33 32 64 61 34 35 62 66 36 30 35 01 01 |1-732da45bf605..| 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3d00f5c86 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 38 31 64 38 65 |..*...$...481d8e| 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 64 38 2d 30 63 31 64 2d 34 66 38 38 2d 38 63 65 |d8-0c1d-4f88-8ce| 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 37 2d 62 62 63 66 64 32 33 36 33 37 36 36 01 01 |7-bbcfd2363766..| 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3d23474f 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 33 31 35 37 33 |..*...$...431573| 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 61 63 2d 38 30 31 34 2d 34 38 36 37 2d 62 37 62 |ac-8014-4867-b7b| 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 65 2d 37 33 38 65 65 35 34 62 38 66 31 35 01 01 |e-738ee54b8f15..| 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3d7ec3591 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 33 36 38 61 30 |..*...$...c368a0| 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 66 66 2d 61 62 65 38 2d 34 65 37 65 2d 62 66 37 |ff-abe8-4e7e-bf7| 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 34 2d 64 33 33 64 61 66 39 66 65 64 33 64 01 01 |4-d33daf9fed3d..| 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3ea48df64 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 38 35 34 63 63 |..*...$...5854cc| 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 31 33 2d 36 38 37 32 2d 34 35 65 36 2d 38 63 38 |13-6872-45e6-8c8| 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 32 2d 36 34 35 63 30 32 38 66 37 31 39 39 01 01 |2-645c028f7199..| 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3f05e87aa 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 62 66 65 31 32 |..*...$...7bfe12| 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 66 39 2d 61 37 36 39 2d 34 66 35 39 2d 62 63 61 |f9-a769-4f59-bca| 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 36 2d 38 63 33 32 61 39 33 66 33 38 30 39 01 01 |6-8c32a93f3809..| 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3f2a64cd4 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 64 32 66 31 61 |..*...$...bd2f1a| 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 65 62 2d 64 36 62 66 2d 34 37 37 37 2d 62 34 64 |eb-d6bf-4777-b4d| 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 65 2d 62 37 64 62 62 61 33 39 64 63 37 62 01 01 |e-b7dbba39dc7b..| 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3f464ab05 2023-12-04T14:36:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 37 39 34 61 32 |..*...$...8794a2| 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 66 37 2d 37 39 31 65 2d 34 64 36 33 2d 61 66 64 |f7-791e-4d63-afd| 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 62 2d 61 31 66 32 39 37 62 33 31 36 66 61 01 01 |b-a1f297b316fa..| 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3fd89eee1 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 64 39 66 66 62 |..*...$...0d9ffb| 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 66 65 2d 35 66 30 62 2d 34 32 36 31 2d 38 32 61 |fe-5f0b-4261-82a| 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 61 2d 64 33 63 36 32 35 64 64 61 31 33 30 01 01 |a-d3c625dda130..| 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3fe9fbf08 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 33 39 33 65 37 |..*...$...e393e7| 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 63 32 2d 33 64 62 65 2d 34 65 35 62 2d 39 35 31 |c2-3dbe-4e5b-951| 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 37 2d 64 61 31 63 37 39 64 35 66 65 34 37 01 01 |7-da1c79d5fe47..| 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_10d3ffbd15f1 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 32 33 33 32 32 36 |..*...$...233226| 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 63 33 2d 30 33 65 62 2d 34 38 62 62 2d 62 63 37 |c3-03eb-48bb-bc7| 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 65 2d 66 65 38 66 35 65 36 62 65 35 61 39 01 01 |e-fe8f5e6be5a9..| 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_11a246f45543 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 39 32 38 30 36 |..*...$...b92806| 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 35 65 2d 63 61 39 62 2d 34 31 61 32 2d 39 65 34 |5e-ca9b-41a2-9e4| 2023-12-04T14:36:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 66 2d 64 62 33 64 38 33 35 31 65 36 33 31 01 01 |f-db3d8351e631..| 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_11a2e0a5aaf0 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (48 bytes) : 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 66 65 33 30 61 |..*...$...9fe30a| 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 37 65 2d 33 39 37 35 2d 34 62 65 38 2d 39 64 64 |7e-3975-4be8-9dd| 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 63 2d 66 66 64 62 66 38 66 37 38 37 65 31 01 01 |c-ffdbf8f787e1..| 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:mirror_mode 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (4 bytes) : 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 00 00 00 |....| 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000004 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:mirror_peer_4d244b58-7457-4b89-a2f7-d69add210e3b 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (132 bytes) : 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 34 64 32 34 34 62 |..~...$...4d244b| 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 35 38 2d 37 34 35 37 2d 34 62 38 39 2d 61 32 66 |58-7457-4b89-a2f| 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 37 2d 64 36 39 61 64 64 32 31 30 65 33 62 08 00 |7-d69add210e3b..| 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 ff ff ff 02 24 00 00 00 66 31 63 38 65 34 35 63 |....$...f1c8e45c| 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 2d 30 62 37 64 2d 34 65 63 35 2d 61 31 66 61 2d |-0b7d-4ec5-a1fa-| 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 62 31 33 39 64 62 38 64 32 64 62 61 d5 e3 6d 65 |b139db8d2dba..me| 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 4b e3 e8 23 |K..#| 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000084 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:mirror_uuid 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 62 33 38 66 39 66 62 64 2d 32 30 65 31 2d 34 39 |b38f9fbd-20e1-49| 2023-12-04T14:36:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 39 66 2d 39 39 64 65 2d 66 62 35 36 32 33 30 33 |9f-99de-fb562303| 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 66 33 36 33 |f363| 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_040b1033-d414-4ab1-bb84-83c330f7ff65_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 d0 53 06 1c 00 |.S...| 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_059e9df9-d50d-4eb9-8306-2dcc0c369e9c_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 c9 39 07 1c 00 |.9...| 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_0d9ffbfe-5f0b-4261-82aa-d3c625dda130_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 8f d4 07 1c 00 |.....| 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_0e39536e-cf2c-4b79-ae6d-a5d1ede7920c_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 52 64 08 1c 00 |Rd...| 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_14168f82-d9d2-41fc-99ed-f1205d8ef156_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 c5 2e 09 1c 00 |.....| 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_1c9b05a7-06ae-420c-ab01-732da45bf605_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 61 b7 09 1c 00 |a....| 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_233226c3-03eb-48bb-bc7e-fe8f5e6be5a9_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 2d 3c 0a 1c 00 |-<...| 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_2a201e15-51f0-492a-8b9c-28ef7a490beb_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 56 c0 0a 1c 00 |V....| 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_431573ac-8014-4867-b7be-738ee54b8f15_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 8e 46 0b 1c 00 |.F...| 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_44871cb2-b60d-4e0e-8e66-9d9bdcbae254_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 77 cb 0b 1c 00 |w....| 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_44e764dc-83c2-44b2-bca6-672e14a04ca5_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 14 53 0c 1c 00 |.S...| 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_481d8ed8-0c1d-4f88-8ce7-bbcfd2363766_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 15 d7 0c 1c 00 |.....| 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_4985d9fc-9c9b-4cb0-a17e-3e72f3e74408_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 d0 5e 0d 1c 00 |.^...| 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_53abed9b-8d5d-4a15-a122-712b8dc9c3d5_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 8b e2 0d 1c 00 |.....| 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_55aa29aa-3f60-443f-a765-b4d1a2257a41_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 39 65 0e 1c 00 |9e...| 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_573c9935-53e1-4562-b5e3-42d4b91a1ca2_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 90 e7 0e 1c 00 |.....| 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_5854cc13-6872-45e6-8c82-645c028f7199_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 42 6a 0f 1c 00 |Bj...| 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_5a31e773-c4a1-4a79-9d13-74cc228e5bc2_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 c0 ef 0f 1c 00 |.....| 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_5f322e9d-b400-4d44-81e0-ffdd216398b2_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 34 78 10 1c 00 |4x...| 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_60498ab7-7310-4365-aa39-30c7019acf9f_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 d6 fd 10 1c 00 |.....| 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_622dfb37-1a24-4c28-8ac0-9f49b29eeead_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 d7 7c 11 1c 00 |.|...| 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_6705f07a-ea33-4f0a-b886-387067319889_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 2d 01 12 1c 00 |-....| 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_709187ee-c3f5-47e9-a97c-a08560690fe3_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 9e 82 12 1c 00 |.....| 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_75662a66-3ee6-422c-bc5b-2ed60a3a9402_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 79 0a 13 1c 00 |y....| 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_75dc1148-0c6b-4bf4-962c-9f4dc15daa97_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 a4 e6 13 1c 00 |.....| 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_7bfe12f9-a769-4f59-bca6-8c32a93f3809_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 8b 82 14 1c 00 |.....| 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_8794a2f7-791e-4d63-afdb-a1f297b316fa_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 37 13 15 1c 00 |7....| 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_90bc0c25-b4ac-4b4a-8f12-bbe6c286b2cd_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 d2 a3 15 1c 00 |.....| 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_99047729-2de2-4c91-90dc-1e2bfe4f2b5f_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 08 36 16 1c 00 |.6...| 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_9d431cf5-1ffc-4498-b3e0-5bba35366d30_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 c5 c0 16 1c 00 |.....| 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_9fe30a7e-3975-4be8-9ddc-ffdbf8f787e1_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 22 4a 17 1c 00 |"J...| 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_ac478d96-184c-4eaa-8c62-f6f0e8288dff_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 9a d6 17 1c 00 |.....| 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_ad2b0660-9c28-48f0-8b26-08f4860ad09e_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 a1 a0 18 1c 00 |.....| 2023-12-04T14:36:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_b6c8de6f-5527-43d7-be14-65c94f60c736_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 a9 2d 19 1c 00 |.-...| 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_b928065e-ca9b-41a2-9e4f-db3d8351e631_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 b4 b3 19 1c 00 |.....| 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_bd2f1aeb-d6bf-4777-b4de-b7dbba39dc7b_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 f2 3a 1a 1c 00 |.:...| 2023-12-04T14:36:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_c368a0ff-abe8-4e7e-bf74-d33daf9fed3d_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 18 c7 1a 1c 00 |.....| 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_c88f1ade-8455-43f9-be6e-45b69b6f15a1_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 a5 4a 1b 1c 00 |.J...| 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_d5429052-81ab-46d6-ab3d-16695051435d_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 fe d2 1b 1c 00 |.....| 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_da0438b0-0666-4c5d-bca2-497b527c3c8a_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 37 56 1c 1c 00 |7V...| 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_e304857c-103b-447a-ae76-3be95494c2ac_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 67 de 1c 1c 00 |g....| 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_e393e7c2-3dbe-4e5b-9517-da1c79d5fe47_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 58 65 1d 1c 00 |Xe...| 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_e7fd68b0-bd04-4c3c-af90-fd5517fef987_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 49 e5 1d 1c 00 |I....| 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_eb8df2f9-471f-4269-a2e3-70777410572c_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 c4 69 1e 1c 00 |.i...| 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_edd4af1f-3c2b-4169-adc5-5d59c2cc5a61_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 00 ef 1e 1c 00 |.....| 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_f19d0e0b-d80d-47ea-9448-f5c84f9d4cc6_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 36 76 1f 1c 00 |6v...| 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_f1ae4816-50e9-48b7-a351-3bb0cf0cd5bb_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 fc fc 1f 1c 00 |.....| 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_f4eea075-92cf-457e-8d7a-77a594f0d72d_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 4a 7e 20 1c 00 |J~ ..| 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_fc8c6534-5c92-48ae-b43e-099c17bc6417_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 10 07 21 1c 00 |..!..| 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:remote_status_global_fe0ec733-207f-4160-afad-a0d4633cee01_f1c8e45c-0b7d-4ec5-a1fa-b139db8d2dba 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (85 bytes) : 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 bc 11 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 36 08 2e bd 10 00 |..........6.....| 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 d5 e3 6d 65 |rting replay..me| 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 8b 21 1c 00 |p.!..| 2023-12-04T14:36:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000055 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_040b1033-d414-4ab1-bb84-83c330f7ff65 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 30 31 37 37 39 32 2e 30 2c |hot":66017792.0,| 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 32 30 33 35 35 38 34 2c |ytes":132035584,| 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 32 30 36 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0206,"remote_sna| 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 32 30 36 2c 22 72 65 70 |:1701700206,"rep| 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 05 20 92 29 00 |"}..me. .).| 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_059e9df9-d50d-4eb9-8306-2dcc0c369e9c 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 39 36 35 37 36 2e 30 2c |hot":67096576.0,| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 31 39 33 31 35 32 2c |ytes":134193152,| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 36 30 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0160,"remote_sna| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 36 30 2c 22 72 65 70 |:1701700160,"rep| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 1f b1 93 29 00 |"}..me...).| 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_0d9ffbfe-5f0b-4261-82aa-d3c625dda130 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 37 31 32 35 37 36 2e 30 2c |hot":66712576.0,| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 33 34 32 35 31 35 32 2c |ytes":133425152,| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 37 38 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0078,"remote_sna| 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 30 37 38 2c 22 72 65 70 |:1701700078,"rep| 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 de 5c 94 29 00 |"}..me.\.).| 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_0e39536e-cf2c-4b79-ae6d-a5d1ede7920c 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 34 37 39 33 36 2e 30 2c |hot":67047936.0,| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 30 39 35 38 37 32 2c |ytes":134095872,| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 30 39 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0109,"remote_sna| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 30 39 2c 22 72 65 70 |:1701700109,"rep| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 f5 04 95 29 00 |"}..me...).| 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_14168f82-d9d2-41fc-99ed-f1205d8ef156 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 34 30 37 36 38 2e 30 2c |hot":67040768.0,| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 30 38 31 35 33 36 2c |ytes":134081536,| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 36 34 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0064,"remote_sna| 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 30 36 34 2c 22 72 65 70 |:1701700064,"rep| 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 37 a6 95 29 00 |"}..me7..).| 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_1c9b05a7-06ae-420c-ab01-732da45bf605 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 39 33 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |99937,"remote_sn| 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 39 33 37 2c 22 72 65 |":1701699937,"re| 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 3b 4f 96 29 00 |e"}..me;O.).| 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_233226c3-03eb-48bb-bc7e-fe8f5e6be5a9 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 38 38 35 36 33 32 2e 30 2c |hot":66885632.0,| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 33 37 37 31 32 36 34 2c |ytes":133771264,| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 32 31 32 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0212,"remote_sna| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 32 31 32 2c 22 72 65 70 |:1701700212,"rep| 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 48 f1 96 29 00 |"}..meH..).| 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_2a201e15-51f0-492a-8b9c-28ef7a490beb 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 34 36 34 37 36 38 2e 30 2c |hot":66464768.0,| 2023-12-04T14:36:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 32 39 32 39 35 33 36 2c |ytes":132929536,| 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 35 33 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0153,"remote_sna| 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 35 33 2c 22 72 65 70 |:1701700153,"rep| 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 df 93 97 29 00 |"}..me...).| 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_431573ac-8014-4867-b7be-738ee54b8f15 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 35 39 34 32 30 31 36 2e 30 2c |hot":65942016.0,| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 31 38 38 34 30 33 32 2c |ytes":131884032,| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 35 38 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0058,"remote_sna| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 30 35 38 2c 22 72 65 70 |:1701700058,"rep| 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 3d 2b 98 29 00 |"}..me=+.).| 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_44871cb2-b60d-4e0e-8e66-9d9bdcbae254 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 34 38 34 34 38 2e 30 2c |hot":67048448.0,| 2023-12-04T14:36:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 30 39 36 38 39 36 2c |ytes":134096896,| 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 38 32 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0182,"remote_sna| 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 38 32 2c 22 72 65 70 |:1701700182,"rep| 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 4e bc 98 29 00 |"}..meN..).| 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_44e764dc-83c2-44b2-bca6-672e14a04ca5 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 35 37 38 31 32 34 38 2e 30 2c |hot":65781248.0,| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 31 35 36 32 34 39 36 2c |ytes":131562496,| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 33 38 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0138,"remote_sna| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 33 38 2c 22 72 65 70 |:1701700138,"rep| 2023-12-04T14:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 b0 59 99 29 00 |"}..me.Y.).| 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_481d8ed8-0c1d-4f88-8ce7-bbcfd2363766 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 34 32 32 37 38 34 2e 30 2c |hot":66422784.0,| 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 32 38 34 35 35 36 38 2c |ytes":132845568,| 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 35 30 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0050,"remote_sna| 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 30 35 30 2c 22 72 65 70 |:1701700050,"rep| 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 a9 f7 99 29 00 |"}..me...).| 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_4985d9fc-9c9b-4cb0-a17e-3e72f3e74408 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 38 37 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |99874,"remote_sn| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 38 37 34 2c 22 72 65 |":1701699874,"re| 2023-12-04T14:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 41 8f 9a 29 00 |e"}..meA..).| 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_53abed9b-8d5d-4a15-a122-712b8dc9c3d5 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 34 33 33 32 38 2e 30 2c |hot":67043328.0,| 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 30 38 36 36 35 36 2c |ytes":134086656,| 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 32 34 32 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0242,"remote_sna| 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 32 34 32 2c 22 72 65 70 |:1701700242,"rep| 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 ab 29 9b 29 00 |"}..me.).).| 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_55aa29aa-3f60-443f-a765-b4d1a2257a41 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 38 37 39 34 38 38 2e 30 2c |hot":66879488.0,| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 33 37 35 38 39 37 36 2c |ytes":133758976,| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 32 32 30 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0220,"remote_sna| 2023-12-04T14:36:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 32 32 30 2c 22 72 65 70 |:1701700220,"rep| 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 38 55 9c 29 00 |"}..me8U.).| 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_573c9935-53e1-4562-b5e3-42d4b91a1ca2 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 31 35 35 30 30 38 2e 30 2c |hot":66155008.0,| 2023-12-04T14:36:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 32 33 31 30 30 31 36 2c |ytes":132310016,| 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 32 36 30 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0260,"remote_sna| 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 32 36 30 2c 22 72 65 70 |:1701700260,"rep| 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 be fa 9c 29 00 |"}..me...).| 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_5854cc13-6872-45e6-8c82-645c028f7199 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 34 36 30 31 36 30 2e 30 2c |hot":66460160.0,| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 32 39 32 30 33 32 30 2c |ytes":132920320,| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 31 37 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0117,"remote_sna| 2023-12-04T14:36:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 31 37 2c 22 72 65 70 |:1701700117,"rep| 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 7f a6 9d 29 00 |"}..me...).| 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_5a31e773-c4a1-4a79-9d13-74cc228e5bc2 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 38 39 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |99894,"remote_sn| 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 38 39 34 2c 22 72 65 |":1701699894,"re| 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 3b 4a 9e 29 00 |e"}..me;J.).| 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_5f322e9d-b400-4d44-81e0-ffdd216398b2 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 35 39 37 33 37 36 2e 30 2c |hot":66597376.0,| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 33 31 39 34 37 35 32 2c |ytes":133194752,| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 39 30 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0090,"remote_sna| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 30 39 30 2c 22 72 65 70 |:1701700090,"rep| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 02 eb 9e 29 00 |"}..me...).| 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_60498ab7-7310-4365-aa39-30c7019acf9f 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 38 39 39 39 36 38 2e 30 2c |hot":66899968.0,| 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 33 37 39 39 39 33 36 2c |ytes":133799936,| 2023-12-04T14:36:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 33 36 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0036,"remote_sna| 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 30 33 36 2c 22 72 65 70 |:1701700036,"rep| 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 42 96 9f 29 00 |"}..meB..).| 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_622dfb37-1a24-4c28-8ac0-9f49b29eeead 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 38 35 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |99852,"remote_sn| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 38 35 32 2c 22 72 65 |":1701699852,"re| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 10 7f a0 29 00 |e"}..me...).| 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_6705f07a-ea33-4f0a-b886-387067319889 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 39 30 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |99907,"remote_sn| 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 39 30 37 2c 22 72 65 |":1701699907,"re| 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 75 16 a1 29 00 |e"}..meu..).| 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_709187ee-c3f5-47e9-a97c-a08560690fe3 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 31 30 37 38 34 30 2e 30 2c |hot":67107840.0,| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 32 31 35 36 38 30 2c |ytes":134215680,| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 32 33 34 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0234,"remote_sna| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 32 33 34 2c 22 72 65 70 |:1701700234,"rep| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 58 aa a1 29 00 |"}..meX..).| 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_75662a66-3ee6-422c-bc5b-2ed60a3a9402 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 39 35 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |99952,"remote_sn| 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 39 35 32 2c 22 72 65 |":1701699952,"re| 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 f0 41 a2 29 00 |e"}..me.A.).| 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_75dc1148-0c6b-4bf4-962c-9f4dc15daa97 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 34 35 34 30 31 36 2e 30 2c |hot":66454016.0,| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 32 39 30 38 30 33 32 2c |ytes":132908032,| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 32 34 38 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0248,"remote_sna| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 32 34 38 2c 22 72 65 70 |:1701700248,"rep| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 42 de a2 29 00 |"}..meB..).| 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_7bfe12f9-a769-4f59-bca6-8c32a93f3809 2023-12-04T14:36:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 38 36 34 36 34 30 2e 30 2c |hot":66864640.0,| 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 33 37 32 39 32 38 30 2c |ytes":133729280,| 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 32 33 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0123,"remote_sna| 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 32 33 2c 22 72 65 70 |:1701700123,"rep| 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 4c 74 a3 29 00 |"}..meLt.).| 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_8794a2f7-791e-4d63-afdb-a1f297b316fa 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 31 33 31 32 30 2e 30 2c |hot":67013120.0,| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 30 32 36 32 34 30 2c |ytes":134026240,| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 30 32 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0102,"remote_sna| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 30 32 2c 22 72 65 70 |:1701700102,"rep| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 c1 08 a4 29 00 |"}..me...).| 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_90bc0c25-b4ac-4b4a-8f12-bbe6c286b2cd 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 35 32 33 36 34 38 2e 30 2c |hot":66523648.0,| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 33 30 34 37 32 39 36 2c |ytes":133047296,| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 34 33 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0043,"remote_sna| 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 30 34 33 2c 22 72 65 70 |:1701700043,"rep| 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 d6 a3 a4 29 00 |"}..me...).| 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_99047729-2de2-4c91-90dc-1e2bfe4f2b5f 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 38 38 31 30 32 34 2e 30 2c |hot":66881024.0,| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 33 37 36 32 30 34 38 2c |ytes":133762048,| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 38 34 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0084,"remote_sna| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 30 38 34 2c 22 72 65 70 |:1701700084,"rep| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 b5 40 a5 29 00 |"}..me.@.).| 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_9d431cf5-1ffc-4498-b3e0-5bba35366d30 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 32 31 33 31 32 2e 30 2c |hot":67021312.0,| 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 30 34 32 36 32 34 2c |ytes":134042624,| 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 33 31 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0131,"remote_sna| 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 33 31 2c 22 72 65 70 |:1701700131,"rep| 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 1e d4 a5 29 00 |"}..me...).| 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_9fe30a7e-3975-4be8-9ddc-ffdbf8f787e1 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (300 bytes) : 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 31 33 33 35 38 38 39 39 32 2e 30 |hot":133588992.0| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 35 38 38 39 39 32 |bytes":133588992| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 |timestamp":17017| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 30 32 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |00022,"remote_sn| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 22 3a 31 37 30 31 37 30 30 30 32 32 2c 22 72 65 |":1701700022,"re| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 65 22 7d bf e3 6d 65 9b 72 a6 29 00 |e"}..me.r.).| 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012c 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_ac478d96-184c-4eaa-8c62-f6f0e8288dff 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 37 36 33 32 36 34 2e 30 2c |hot":66763264.0,| 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 33 35 32 36 35 32 38 2c |ytes":133526528,| 2023-12-04T14:36:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 39 36 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0096,"remote_sna| 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 30 39 36 2c 22 72 65 70 |:1701700096,"rep| 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 8a 15 a7 29 00 |"}..me...).| 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_ad2b0660-9c28-48f0-8b26-08f4860ad09e 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 30 36 34 36 34 2e 30 2c |hot":67006464.0,| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 30 31 32 39 32 38 2c |ytes":134012928,| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 32 30 30 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0200,"remote_sna| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 32 30 30 2c 22 72 65 70 |:1701700200,"rep| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 d0 b5 a7 29 00 |"}..me...).| 2023-12-04T14:36:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_b6c8de6f-5527-43d7-be14-65c94f60c736 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 39 36 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |99969,"remote_sn| 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 39 36 39 2c 22 72 65 |":1701699969,"re| 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 21 5e a8 29 00 |e"}..me!^.).| 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_b928065e-ca9b-41a2-9e4f-db3d8351e631 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (300 bytes) : 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 31 33 33 39 39 36 35 34 34 2e 30 |hot":133996544.0| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 39 39 36 35 34 34 |bytes":133996544| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 |timestamp":17017| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 30 31 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |00016,"remote_sn| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 22 3a 31 37 30 31 37 30 30 30 31 36 2c 22 72 65 |":1701700016,"re| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 65 22 7d bf e3 6d 65 fc fa a8 29 00 |e"}..me...).| 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012c 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_bd2f1aeb-d6bf-4777-b4de-b7dbba39dc7b 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 39 31 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |99915,"remote_sn| 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 39 31 35 2c 22 72 65 |":1701699915,"re| 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 7f 8c a9 29 00 |e"}..me...).| 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_c368a0ff-abe8-4e7e-bf74-d33daf9fed3d 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 39 36 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |99963,"remote_sn| 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 39 36 33 2c 22 72 65 |":1701699963,"re| 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 67 24 aa 29 00 |e"}..meg$.).| 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_c88f1ade-8455-43f9-be6e-45b69b6f15a1 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 32 34 38 39 36 2e 30 2c |hot":67024896.0,| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 30 34 39 37 39 32 2c |ytes":134049792,| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 32 35 34 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0254,"remote_sna| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 32 35 34 2c 22 72 65 70 |:1701700254,"rep| 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 7f 19 ab 29 00 |"}..me...).| 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_d5429052-81ab-46d6-ab3d-16695051435d 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 35 36 36 34 30 2e 30 2c |hot":67056640.0,| 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 31 31 33 32 38 30 2c |ytes":134113280,| 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 32 39 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0029,"remote_sna| 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 30 32 39 2c 22 72 65 70 |:1701700029,"rep| 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 66 c7 ab 29 00 |"}..mef..).| 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_da0438b0-0666-4c5d-bca2-497b527c3c8a 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 35 39 32 36 36 35 36 2e 30 2c |hot":65926656.0,| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 31 38 35 33 33 31 32 2c |ytes":131853312,| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 32 32 38 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0228,"remote_sna| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 32 32 38 2c 22 72 65 70 |:1701700228,"rep| 2023-12-04T14:36:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 d3 6d ac 29 00 |"}..me.m.).| 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_e304857c-103b-447a-ae76-3be95494c2ac 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 37 32 30 30 30 2e 30 2c |hot":67072000.0,| 2023-12-04T14:36:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 31 34 34 30 30 30 2c |ytes":134144000,| 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 36 38 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0168,"remote_sna| 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 36 38 2c 22 72 65 70 |:1701700168,"rep| 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 de 16 ad 29 00 |"}..me...).| 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_e393e7c2-3dbe-4e5b-9517-da1c79d5fe47 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 36 33 30 33 34 38 38 2e 30 2c |hot":66303488.0,| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 32 36 30 36 39 37 36 2c |ytes":132606976,| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 39 34 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0194,"remote_sna| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 39 34 2c 22 72 65 70 |:1701700194,"rep| 2023-12-04T14:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 d6 bf ad 29 00 |"}..me...).| 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_e7fd68b0-bd04-4c3c-af90-fd5517fef987 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 35 39 31 37 39 35 32 2e 30 2c |hot":65917952.0,| 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 31 38 33 35 39 30 34 2c |ytes":131835904,| 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 34 36 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0146,"remote_sna| 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 34 36 2c 22 72 65 70 |:1701700146,"rep| 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 30 5b ae 29 00 |"}..me0[.).| 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_eb8df2f9-471f-4269-a2e3-70777410572c 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 37 37 31 32 30 2e 30 2c |hot":67077120.0,| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 31 35 34 32 34 30 2c |ytes":134154240,| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 37 34 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0174,"remote_sna| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 37 34 2c 22 72 65 70 |:1701700174,"rep| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 84 42 af 29 00 |"}..me.B.).| 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_edd4af1f-3c2b-4169-adc5-5d59c2cc5a61 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 30 37 37 31 32 30 2e 30 2c |hot":67077120.0,| 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 31 35 34 32 34 30 2c |ytes":134154240,| 2023-12-04T14:36:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 31 38 38 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0188,"remote_sna| 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 31 38 38 2c 22 72 65 70 |:1701700188,"rep| 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 1f df af 29 00 |"}..me...).| 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_f19d0e0b-d80d-47ea-9448-f5c84f9d4cc6 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 38 34 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |99844,"remote_sn| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 38 34 34 2c 22 72 65 |":1701699844,"re| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 26 7d b0 29 00 |e"}..me&}.).| 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_f1ae4816-50e9-48b7-a351-3bb0cf0cd5bb 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 38 36 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |99867,"remote_sn| 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 38 36 37 2c 22 72 65 |":1701699867,"re| 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 85 1c b1 29 00 |e"}..me...).| 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_f4eea075-92cf-457e-8d7a-77a594f0d72d 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 39 32 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |99923,"remote_sn| 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 39 32 33 2c 22 72 65 |":1701699923,"re| 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 36 aa b1 29 00 |e"}..me6..).| 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_fc8c6534-5c92-48ae-b43e-099c17bc6417 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (299 bytes) : 2023-12-04T14:36:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 f3 00 00 00 04 e5 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 36 37 31 30 37 38 34 30 2e 30 2c |hot":67107840.0,| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 62 |"last_snapshot_b| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 79 74 65 73 22 3a 31 33 34 32 31 35 36 38 30 2c |ytes":134215680,| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f 73 |"last_snapshot_s| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c 22 |ync_seconds":0,"| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f 74 |local_snapshot_t| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 37 30 |imestamp":170170| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 30 30 37 31 2c 22 72 65 6d 6f 74 65 5f 73 6e 61 |0071,"remote_sna| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 22 |pshot_timestamp"| 2023-12-04T14:36:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 3a 31 37 30 31 37 30 30 30 37 31 2c 22 72 65 70 |:1701700071,"rep| 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c 65 |lay_state":"idle| 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000120 22 7d bf e3 6d 65 34 47 b2 29 00 |"}..me4G.).| 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000012b 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:status_global_fe0ec733-207f-4160-afad-a0d4633cee01 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (284 bytes) : 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 2c 00 00 00 08 d3 10 00 00 00 00 00 00 01 |..,.............| 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 6c 69 bc 29 10 00 |..........li.)..| 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 02 00 00 00 ac 15 0f b0 00 00 00 00 00 00 |................| 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2023-12-04T14:36:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2023-12-04T14:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2023-12-04T14:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2023-12-04T14:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 30 31 36 |timestamp":17016| 2023-12-04T14:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000d0 39 39 38 38 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |99882,"remote_sn| 2023-12-04T14:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2023-12-04T14:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:000000f0 22 3a 31 37 30 31 36 39 39 38 38 32 2c 22 72 65 |":1701699882,"re| 2023-12-04T14:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2023-12-04T14:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000110 65 22 7d bf e3 6d 65 46 e2 b2 29 00 |e"}..meF..).| 2023-12-04T14:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:0000011c 2023-12-04T14:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror rbd_mirror_leader omap vals' 2023-12-04T14:36:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror rbd_mirror_leader omap vals 2023-12-04T14:36:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirror_leader 2023-12-04T14:36:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_040b1033-d414-4ab1-bb84-83c330f7ff65 2023-12-04T14:36:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 ae e0 |..........4307..| 2023-12-04T14:36:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 3d 29 1c 38 0a 00 00 00 01 01 04 00 00 00 |me=).8..........| 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_059e9df9-d50d-4eb9-8306-2dcc0c369e9c 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 58 e0 |..........4307X.| 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 4b 28 c5 01 0a 00 00 00 01 01 04 00 00 00 |meK(............| 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_0d9ffbfe-5f0b-4261-82aa-d3c625dda130 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 d9 df |..........4307..| 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 1a 0d e2 02 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_0e39536e-cf2c-4b79-ae6d-a5d1ede7920c 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 0f e0 |..........4307..| 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 f0 67 ed 01 0a 00 00 00 01 01 04 00 00 00 |me.g............| 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_14168f82-d9d2-41fc-99ed-f1205d8ef156 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 c1 df |..........4307..| 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 9e 2d 35 00 0a 00 00 00 01 01 04 00 00 00 |me.-5...........| 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_1c9b05a7-06ae-420c-ab01-732da45bf605 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 62 e1 |..........4307b.| 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 7d 8e 81 3b 0a 00 00 00 01 01 04 00 00 00 |me}..;..........| 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_233226c3-03eb-48bb-bc7e-fe8f5e6be5a9 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 b5 e0 |..........4307..| 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 7c 1f 64 03 0a 00 00 00 01 01 04 00 00 00 |me|.d...........| 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_2a201e15-51f0-492a-8b9c-28ef7a490beb 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 4e e0 |..........4307N.| 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 22 9f 89 03 0a 00 00 00 01 01 04 00 00 00 |me".............| 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_431573ac-8014-4867-b7be-738ee54b8f15 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 bb df |..........4307..| 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 5c 63 f6 23 0a 00 00 00 01 01 04 00 00 00 |me\c.#..........| 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_44871cb2-b60d-4e0e-8e66-9d9bdcbae254 2023-12-04T14:36:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 81 e0 |..........4307..| 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 4e e6 ee 02 0a 00 00 00 01 01 04 00 00 00 |meN.............| 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_44e764dc-83c2-44b2-bca6-672e14a04ca5 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 3f e0 |..........4307?.| 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 35 12 fb 2b 0a 00 00 00 01 01 04 00 00 00 |me5..+..........| 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_481d8ed8-0c1d-4f88-8ce7-bbcfd2363766 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 b1 df |..........4307..| 2023-12-04T14:36:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 99 5f de 24 0a 00 00 00 01 01 04 00 00 00 |me._.$..........| 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_4985d9fc-9c9b-4cb0-a17e-3e72f3e74408 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 23 e1 |..........4307#.| 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 4c ec 8f 06 0a 00 00 00 01 01 04 00 00 00 |meL.............| 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_53abed9b-8d5d-4a15-a122-712b8dc9c3d5 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 dc e0 |..........4307..| 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 00 d2 12 06 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_55aa29aa-3f60-443f-a765-b4d1a2257a41 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 be e0 |..........4307..| 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 8d 6b 43 08 0a 00 00 00 01 01 04 00 00 00 |me.kC...........| 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_573c9935-53e1-4562-b5e3-42d4b91a1ca2 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 fc e0 |..........4307..| 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 18 95 aa 38 0a 00 00 00 01 01 04 00 00 00 |me...8..........| 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_5854cc13-6872-45e6-8c82-645c028f7199 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 17 e0 |..........4307..| 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 de ce 4e 28 0a 00 00 00 01 01 04 00 00 00 |me..N(..........| 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_5a31e773-c4a1-4a79-9d13-74cc228e5bc2 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 37 e1 |..........43077.| 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 22 72 dc 39 0a 00 00 00 01 01 04 00 00 00 |me"r.9..........| 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_5f322e9d-b400-4d44-81e0-ffdd216398b2 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 ec df |..........4307..| 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 3a e4 1f 0d 0a 00 00 00 01 01 04 00 00 00 |me:.............| 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_60498ab7-7310-4365-aa39-30c7019acf9f 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 9d df |..........4307..| 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 a6 1b 07 03 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_622dfb37-1a24-4c28-8ac0-9f49b29eeead 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 0d e1 |..........4307..| 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 b1 da df 0b 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_6705f07a-ea33-4f0a-b886-387067319889 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 45 e1 |..........4307E.| 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 cb 15 23 0d 0a 00 00 00 01 01 04 00 00 00 |me..#...........| 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_709187ee-c3f5-47e9-a97c-a08560690fe3 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 ce e0 |..........4307..| 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 dc fe be 02 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_75662a66-3ee6-422c-bc5b-2ed60a3a9402 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 71 e1 |..........4307q.| 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 b0 24 7d 08 0a 00 00 00 01 01 04 00 00 00 |me.$}...........| 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_75dc1148-0c6b-4bf4-962c-9f4dc15daa97 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 e8 e0 |..........4307..| 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 dd 54 eb 36 0a 00 00 00 01 01 04 00 00 00 |me.T.6..........| 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_7bfe12f9-a769-4f59-bca6-8c32a93f3809 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 24 e0 |..........4307$.| 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 20 c1 ae 29 0a 00 00 00 01 01 04 00 00 00 |me ..)..........| 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_8794a2f7-791e-4d63-afdb-a1f297b316fa 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 ff df |..........4307..| 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 75 d4 77 3b 0a 00 00 00 01 01 04 00 00 00 |meu.w;..........| 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_90bc0c25-b4ac-4b4a-8f12-bbe6c286b2cd 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 a8 df |..........4307..| 2023-12-04T14:36:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 73 19 8e 24 0a 00 00 00 01 01 04 00 00 00 |mes..$..........| 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_99047729-2de2-4c91-90dc-1e2bfe4f2b5f 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 e2 df |..........4307..| 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 02 9e 0d 08 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_9d431cf5-1ffc-4498-b3e0-5bba35366d30 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 32 e0 |..........43072.| 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 f9 49 75 02 0a 00 00 00 01 01 04 00 00 00 |me.Iu...........| 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_9fe30a7e-3975-4be8-9ddc-ffdbf8f787e1 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 91 df |..........4307..| 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 c9 93 00 0f 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_ac478d96-184c-4eaa-8c62-f6f0e8288dff 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 f8 df |..........4307..| 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 d9 34 37 27 0a 00 00 00 01 01 04 00 00 00 |me.47'..........| 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_ad2b0660-9c28-48f0-8b26-08f4860ad09e 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 a0 e0 |..........4307..| 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 e9 38 c1 06 0a 00 00 00 01 01 04 00 00 00 |me.8............| 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_b6c8de6f-5527-43d7-be14-65c94f60c736 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 82 e1 |..........4307..| 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 85 55 fd 10 0a 00 00 00 01 01 04 00 00 00 |me.U............| 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_b928065e-ca9b-41a2-9e4f-db3d8351e631 2023-12-04T14:36:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 91 df |..........4307..| 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 9b a6 00 0f 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_bd2f1aeb-d6bf-4777-b4de-b7dbba39dc7b 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 4c e1 |..........4307L.| 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 4d d1 6a 3b 0a 00 00 00 01 01 04 00 00 00 |meM.j;..........| 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_c368a0ff-abe8-4e7e-bf74-d33daf9fed3d 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 7c e1 |..........4307|.| 2023-12-04T14:36:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 04 fd ee 0e 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_c88f1ade-8455-43f9-be6e-45b69b6f15a1 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 f0 e0 |..........4307..| 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 a6 96 93 07 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_d5429052-81ab-46d6-ab3d-16695051435d 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 91 df |..........4307..| 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 e4 b1 00 0f 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_da0438b0-0666-4c5d-bca2-497b527c3c8a 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 c4 e0 |..........4307..| 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 6e 7b 55 09 0a 00 00 00 01 01 04 00 00 00 |men{U...........| 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_e304857c-103b-447a-ae76-3be95494c2ac 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 65 e0 |..........4307e.| 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 80 3a c2 2c 0a 00 00 00 01 01 04 00 00 00 |me.:.,..........| 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_e393e7c2-3dbe-4e5b-9517-da1c79d5fe47 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 94 e0 |..........4307..| 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 34 2a 44 03 0a 00 00 00 01 01 04 00 00 00 |me4*D...........| 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_e7fd68b0-bd04-4c3c-af90-fd5517fef987 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 47 e0 |..........4307G.| 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 a6 61 a2 2c 0a 00 00 00 01 01 04 00 00 00 |me.a.,..........| 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_eb8df2f9-471f-4269-a2e3-70777410572c 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 72 e0 |..........4307r.| 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 80 51 47 39 0a 00 00 00 01 01 04 00 00 00 |me.QG9..........| 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_edd4af1f-3c2b-4169-adc5-5d59c2cc5a61 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 8e e0 |..........4307..| 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 64 10 84 30 0a 00 00 00 01 01 04 00 00 00 |med..0..........| 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_f19d0e0b-d80d-47ea-9448-f5c84f9d4cc6 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 05 e1 |..........4307..| 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 e3 1c 8f 06 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_f1ae4816-50e9-48b7-a351-3bb0cf0cd5bb 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 1c e1 |..........4307..| 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 73 6a 6b 04 0a 00 00 00 01 01 04 00 00 00 |mesjk...........| 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_f4eea075-92cf-457e-8d7a-77a594f0d72d 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 54 e1 |..........4307T.| 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 d2 7c a5 05 0a 00 00 00 01 01 04 00 00 00 |me.|............| 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_fc8c6534-5c92-48ae-b43e-099c17bc6417 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 ca df |..........4307..| 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 09 63 ed 3a 0a 00 00 00 01 01 04 00 00 00 |me.c.:..........| 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image_map_fe0ec733-207f-4160-afad-a0d4633cee01 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 33 30 37 2b e1 |..........4307+.| 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 6d 65 f2 c4 d5 0b 0a 00 00 00 01 01 04 00 00 00 |me..............| 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 00 00 00 00 |....| 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_4307 2023-12-04T14:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_4786 2023-12-04T14:36:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_4884 2023-12-04T14:36:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_5329 2023-12-04T14:36:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-04T14:36:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror ns1 images' 2023-12-04T14:36:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror ns1 images 2023-12-04T14:36:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 ls -l 2023-12-04T14:36:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirrorns1 mirror pool info 2023-12-04T14:36:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirrorns1 mirror pool info' 2023-12-04T14:36:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool info 2023-12-04T14:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mode: image 2023-12-04T14:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirrorns1 mirror pool status 2023-12-04T14:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirrorns1 mirror pool status' 2023-12-04T14:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool status --verbose 2023-12-04T14:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:health: OK 2023-12-04T14:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:daemon health: OK 2023-12-04T14:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image health: OK 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:images: 0 total 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:DAEMONS 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 4258: 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 4307 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.3 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: true 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 4783: 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 4786 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.2 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: false 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 4881: 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 4884 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.1 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: false 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 5326: 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 5329 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.0 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: false 2023-12-04T14:36:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:IMAGES 2023-12-04T14:36:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 ls 2023-12-04T14:36:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror ns1 rbd_mirroring omap vals 2023-12-04T14:36:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror ns1 rbd_mirroring omap vals' 2023-12-04T14:36:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirroring 2023-12-04T14:36:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:mirror_mode 2023-12-04T14:36:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (4 bytes) : 2023-12-04T14:36:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 00 00 00 |....| 2023-12-04T14:36:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000004 2023-12-04T14:36:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:mirror_uuid 2023-12-04T14:36:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 31 64 39 36 36 62 32 36 2d 31 64 61 32 2d 34 37 |1d966b26-1da2-47| 2023-12-04T14:36:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 36 32 2d 61 39 34 30 2d 30 38 37 36 31 37 63 34 |62-a940-087617c4| 2023-12-04T14:36:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 31 61 63 35 |1ac5| 2023-12-04T14:36:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror ns1 rbd_mirror_leader omap vals' 2023-12-04T14:36:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror ns1 rbd_mirror_leader omap vals 2023-12-04T14:36:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2023-12-04T14:36:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_4307 2023-12-04T14:36:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_4884 2023-12-04T14:36:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_5329 2023-12-04T14:36:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.994 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-04T14:36:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror ns2 images 2023-12-04T14:36:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror ns2 images' 2023-12-04T14:36:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 ls -l 2023-12-04T14:36:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirrorns2 mirror pool info 2023-12-04T14:36:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirrorns2 mirror pool info' 2023-12-04T14:36:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool info 2023-12-04T14:36:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mode: image 2023-12-04T14:36:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirrorns2 mirror pool status 2023-12-04T14:36:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirrorns2 mirror pool status' 2023-12-04T14:36:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:36:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool status --verbose 2023-12-04T14:36:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:health: OK 2023-12-04T14:36:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:daemon health: OK 2023-12-04T14:36:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image health: OK 2023-12-04T14:36:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:images: 0 total 2023-12-04T14:36:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:DAEMONS 2023-12-04T14:36:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 4258: 2023-12-04T14:36:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 4307 2023-12-04T14:36:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.3 2023-12-04T14:36:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: true 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 4783: 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 4786 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.2 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: false 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 4881: 2023-12-04T14:36:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 4884 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.1 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: false 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 5326: 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 5329 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.0 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: false 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:IMAGES 2023-12-04T14:36:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 ls 2023-12-04T14:36:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror ns2 rbd_mirroring omap vals 2023-12-04T14:36:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror ns2 rbd_mirroring omap vals' 2023-12-04T14:36:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirroring 2023-12-04T14:36:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:mirror_mode 2023-12-04T14:36:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (4 bytes) : 2023-12-04T14:36:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 00 00 00 |....| 2023-12-04T14:36:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000004 2023-12-04T14:36:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:mirror_uuid 2023-12-04T14:36:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 63 38 34 33 64 39 38 33 2d 34 34 65 65 2d 34 62 |c843d983-44ee-4b| 2023-12-04T14:36:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 34 66 2d 62 32 34 66 2d 35 65 64 65 35 32 32 30 |4f-b24f-5ede5220| 2023-12-04T14:36:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 63 65 30 33 |ce03| 2023-12-04T14:36:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror ns2 rbd_mirror_leader omap vals' 2023-12-04T14:36:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror ns2 rbd_mirror_leader omap vals 2023-12-04T14:36:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2023-12-04T14:36:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_4307 2023-12-04T14:36:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_4884 2023-12-04T14:36:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_5329 2023-12-04T14:36:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-04T14:36:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-04T14:36:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parent images 2023-12-04T14:36:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parent images' 2023-12-04T14:36:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' ls -l 2023-12-04T14:36:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parent mirror pool info 2023-12-04T14:36:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parent mirror pool info' 2023-12-04T14:36:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool info 2023-12-04T14:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mode: image 2023-12-04T14:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Site Name: cluster1 2023-12-04T14:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Peer Sites: 2023-12-04T14:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:UUID: 4d9db1f6-5ae6-4138-ab81-f317a659b2a2 2023-12-04T14:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Name: cluster2 2023-12-04T14:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Mirror UUID: 1151b25f-4f36-4138-a84a-36555f27da61 2023-12-04T14:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Direction: rx-tx 2023-12-04T14:36:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:Client: client.mirror 2023-12-04T14:36:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parent mirror pool status 2023-12-04T14:36:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parent mirror pool status' 2023-12-04T14:36:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:36:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool status --verbose 2023-12-04T14:36:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:health: OK 2023-12-04T14:36:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:daemon health: OK 2023-12-04T14:36:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:image health: OK 2023-12-04T14:36:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:images: 0 total 2023-12-04T14:36:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:DAEMONS 2023-12-04T14:36:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 4258: 2023-12-04T14:36:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 4310 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.3 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: true 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 4783: 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 4790 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.2 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: false 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 4881: 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 4890 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.1 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: false 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:service 5326: 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: instance_id: 5331 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: client_id: mirror.0 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: hostname: smithi176 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: version: 18.0.0-7672-g7c730b29 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: leader: false 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: health: OK 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:IMAGES 2023-12-04T14:36:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' ls 2023-12-04T14:36:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parent rbd_mirroring omap vals 2023-12-04T14:36:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parent rbd_mirroring omap vals' 2023-12-04T14:36:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2023-12-04T14:36:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:mirror_mode 2023-12-04T14:36:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (4 bytes) : 2023-12-04T14:36:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 01 00 00 00 |....| 2023-12-04T14:36:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000004 2023-12-04T14:36:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:mirror_peer_4d9db1f6-5ae6-4138-ab81-f317a659b2a2 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (132 bytes) : 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 34 64 39 64 62 31 |..~...$...4d9db1| 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 66 36 2d 35 61 65 36 2d 34 31 33 38 2d 61 62 38 |f6-5ae6-4138-ab8| 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 31 2d 66 33 31 37 61 36 35 39 62 32 61 32 08 00 |1-f317a659b2a2..| 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000050 ff ff ff 02 24 00 00 00 31 31 35 31 62 32 35 66 |....$...1151b25f| 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000060 2d 34 66 33 36 2d 34 31 33 38 2d 61 38 34 61 2d |-4f36-4138-a84a-| 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000070 33 36 35 35 35 66 32 37 64 61 36 31 d5 e3 6d 65 |36555f27da61..me| 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000080 e4 1e c7 2c |...,| 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000084 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:mirror_uuid 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (36 bytes) : 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000000 65 63 38 65 38 63 65 32 2d 65 31 62 38 2d 34 31 |ec8e8ce2-e1b8-41| 2023-12-04T14:36:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000010 36 66 2d 61 65 34 32 2d 64 36 33 31 37 39 64 30 |6f-ae42-d63179d0| 2023-12-04T14:36:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000020 61 62 39 65 |ab9e| 2023-12-04T14:36:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:00000024 2023-12-04T14:36:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parent rbd_mirror_leader omap vals' 2023-12-04T14:36:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parent rbd_mirror_leader omap vals 2023-12-04T14:36:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2023-12-04T14:36:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_4310 2023-12-04T14:36:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_4890 2023-12-04T14:36:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:instance_5331 2023-12-04T14:36:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:value (0 bytes) : 2023-12-04T14:36:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-04T14:36:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parent ns1 images' 2023-12-04T14:36:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parent ns1 images 2023-12-04T14:36:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls -l 2023-12-04T14:36:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: namespace 'ns1' does not exist. 2023-12-04T14:36:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: listing images failed: (2) No such file or directory 2023-12-04T14:36:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parentns1 mirror pool info 2023-12-04T14:36:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parentns1 mirror pool info' 2023-12-04T14:36:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool info 2023-12-04T14:36:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: namespace 'ns1' does not exist. 2023-12-04T14:36:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parentns1 mirror pool status' 2023-12-04T14:36:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parentns1 mirror pool status 2023-12-04T14:36:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:36:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool status --verbose 2023-12-04T14:36:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: namespace 'ns1' does not exist. 2023-12-04T14:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls 2023-12-04T14:36:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parent ns1 rbd_mirroring omap vals 2023-12-04T14:36:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parent ns1 rbd_mirroring omap vals' 2023-12-04T14:36:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2023-12-04T14:36:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2023-12-04T14:36:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parent ns1 rbd_mirror_leader omap vals' 2023-12-04T14:36:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parent ns1 rbd_mirror_leader omap vals 2023-12-04T14:36:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2023-12-04T14:36:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:error getting omap keys mirror_parent/rbd_mirror_leader: (2) No such file or directory 2023-12-04T14:36:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-04T14:36:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parent ns2 images 2023-12-04T14:36:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parent ns2 images' 2023-12-04T14:36:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls -l 2023-12-04T14:36:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: namespace 'ns2' does not exist. 2023-12-04T14:36:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: listing images failed: (2) No such file or directory 2023-12-04T14:36:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parentns2 mirror pool info 2023-12-04T14:36:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parentns2 mirror pool info' 2023-12-04T14:36:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool info 2023-12-04T14:36:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: namespace 'ns2' does not exist. 2023-12-04T14:36:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parentns2 mirror pool status 2023-12-04T14:36:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parentns2 mirror pool status' 2023-12-04T14:36:12.696 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:36:12.696 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool status --verbose 2023-12-04T14:36:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: namespace 'ns2' does not exist. 2023-12-04T14:36:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls 2023-12-04T14:36:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parent ns2 rbd_mirroring omap vals 2023-12-04T14:36:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parent ns2 rbd_mirroring omap vals' 2023-12-04T14:36:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2023-12-04T14:36:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2023-12-04T14:36:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 mirror_parent ns2 rbd_mirror_leader omap vals' 2023-12-04T14:36:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 mirror_parent ns2 rbd_mirror_leader omap vals 2023-12-04T14:36:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2023-12-04T14:36:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:error getting omap keys mirror_parent/rbd_mirror_leader: (2) No such file or directory 2023-12-04T14:36:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:36:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:36:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2023-12-04T14:36:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 status 2023-12-04T14:36:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 status' 2023-12-04T14:36:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:36:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster2 -s 2023-12-04T14:41:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:2023-12-04T14:41:13.103+0000 7f97e1ce7640 0 monclient(hunting): authenticate timed out after 300 2023-12-04T14:41:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-04T14:41:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:41:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster2 service dump 2023-12-04T14:42:19.894 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster1.client.mirror.2 2023-12-04T14:42:19.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 6 2023-12-04T14:42:19.895 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-04T14:42:19.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr:*** Caught signal (Aborted) ** 2023-12-04T14:42:19.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: in thread 7f7d51a0d640 thread_name:rbd-mirror 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f7d51854db0] 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f7d5189c39a] 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 3: pthread_cond_wait() 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f7d51cd5500] 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 5: rbd-mirror(+0x339cdb) [0x561283832cdb] 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x56128383c7b3] 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x56128383cb10] 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 8: rbd-mirror(+0x92f538) [0x561283e28538] 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 9: rbd-mirror(+0x92f520) [0x561283e28520] 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x56128382bc65] 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 11: main() 2023-12-04T14:42:19.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f7d5183feb0] 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 13: __libc_start_main() 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 14: _start() 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr:2023-12-04T14:42:19.895+0000 7f7d51a0d640 -1 *** Caught signal (Aborted) ** 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: in thread 7f7d51a0d640 thread_name:rbd-mirror 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f7d51854db0] 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f7d5189c39a] 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 3: pthread_cond_wait() 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f7d51cd5500] 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 5: rbd-mirror(+0x339cdb) [0x561283832cdb] 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x56128383c7b3] 2023-12-04T14:42:19.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x56128383cb10] 2023-12-04T14:42:19.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 8: rbd-mirror(+0x92f538) [0x561283e28538] 2023-12-04T14:42:19.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 9: rbd-mirror(+0x92f520) [0x561283e28520] 2023-12-04T14:42:19.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x56128382bc65] 2023-12-04T14:42:19.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 11: main() 2023-12-04T14:42:19.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f7d5183feb0] 2023-12-04T14:42:19.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 13: __libc_start_main() 2023-12-04T14:42:19.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 14: _start() 2023-12-04T14:42:19.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T14:42:19.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 2023-12-04T14:42:19.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: -2763> 2023-12-04T14:32:10.756+0000 7f7d4e4d1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 103931) UID: 1000 2023-12-04T14:42:19.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 0> 2023-12-04T14:42:19.895+0000 7f7d51a0d640 -1 *** Caught signal (Aborted) ** 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: in thread 7f7d51a0d640 thread_name:rbd-mirror 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f7d51854db0] 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f7d5189c39a] 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 3: pthread_cond_wait() 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f7d51cd5500] 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 5: rbd-mirror(+0x339cdb) [0x561283832cdb] 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x56128383c7b3] 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x56128383cb10] 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 8: rbd-mirror(+0x92f538) [0x561283e28538] 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 9: rbd-mirror(+0x92f520) [0x561283e28520] 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x56128382bc65] 2023-12-04T14:42:19.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 11: main() 2023-12-04T14:42:19.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f7d5183feb0] 2023-12-04T14:42:19.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 13: __libc_start_main() 2023-12-04T14:42:19.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 14: _start() 2023-12-04T14:42:19.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T14:42:19.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 2023-12-04T14:42:19.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: -6884> 2023-12-04T14:32:10.756+0000 7f7d4e4d1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 103931) UID: 1000 2023-12-04T14:42:19.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: -6883> 2023-12-04T14:42:19.895+0000 7f7d51a0d640 -1 *** Caught signal (Aborted) ** 2023-12-04T14:42:19.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: in thread 7f7d51a0d640 thread_name:rbd-mirror 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f7d51854db0] 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f7d5189c39a] 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 3: pthread_cond_wait() 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f7d51cd5500] 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 5: rbd-mirror(+0x339cdb) [0x561283832cdb] 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x103) [0x56128383c7b3] 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x56128383cb10] 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 8: rbd-mirror(+0x92f538) [0x561283e28538] 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 9: rbd-mirror(+0x92f520) [0x561283e28520] 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x56128382bc65] 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 11: main() 2023-12-04T14:42:19.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 12: /lib64/libc.so.6(+0x3feb0) [0x7f7d5183feb0] 2023-12-04T14:42:19.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 13: __libc_start_main() 2023-12-04T14:42:19.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 14: _start() 2023-12-04T14:42:19.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T14:42:19.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr: 2023-12-04T14:42:20.067 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2023-12-04T14:42:20.067 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-04T14:42:20.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi176.stderr:daemon-helper: command crashed with signal 6 2023-12-04T14:42:20.570 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2023-12-04T14:42:20.570 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2023-12-04T14:42:20.571 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2023-12-04T14:42:20.571 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2023-12-04T14:42:20.571 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2023-12-04T14:42:20.572 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:42:20.570+0000 7f78c6f9b640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 85269) UID: 0 2023-12-04T14:42:20.572 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:42:20.570+0000 7f78c6f9b640 -1 osd.0 281 *** Got signal Terminated *** 2023-12-04T14:42:20.572 INFO:tasks.ceph.cluster1.osd.0.smithi146.stderr:2023-12-04T14:42:20.570+0000 7f78c6f9b640 -1 osd.0 281 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T14:42:20.572 INFO:tasks.ceph.cluster1.osd.1.smithi146.stderr:2023-12-04T14:42:20.570+0000 7f0c3364d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 85272) UID: 0 2023-12-04T14:42:20.572 INFO:tasks.ceph.cluster1.osd.1.smithi146.stderr:2023-12-04T14:42:20.570+0000 7f0c3364d640 -1 osd.1 281 *** Got signal Terminated *** 2023-12-04T14:42:20.572 INFO:tasks.ceph.cluster1.osd.1.smithi146.stderr:2023-12-04T14:42:20.570+0000 7f0c3364d640 -1 osd.1 281 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T14:42:20.573 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:42:20.570+0000 7f5191779640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 85274) UID: 0 2023-12-04T14:42:20.573 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:42:20.570+0000 7f5191779640 -1 osd.2 281 *** Got signal Terminated *** 2023-12-04T14:42:20.573 INFO:tasks.ceph.cluster1.osd.2.smithi146.stderr:2023-12-04T14:42:20.570+0000 7f5191779640 -1 osd.2 281 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-04T14:42:20.573 INFO:tasks.ceph.cluster1.mon.a.smithi146.stderr:2023-12-04T14:42:20.570+0000 7f762b61d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 84937) UID: 0 2023-12-04T14:42:20.573 INFO:tasks.ceph.cluster1.mon.a.smithi146.stderr:2023-12-04T14:42:20.570+0000 7f762b61d640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-04T14:42:20.573 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:42:20.571+0000 7f084593a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 84939) UID: 0 2023-12-04T14:42:20.573 INFO:tasks.ceph.cluster1.mgr.x.smithi146.stderr:2023-12-04T14:42:20.571+0000 7f084593a640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-04T14:42:40.482 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.1 2023-12-04T14:42:40.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 6 2023-12-04T14:42:40.483 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-04T14:42:40.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr:*** Caught signal (Aborted) ** 2023-12-04T14:42:40.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: in thread 7f0841c1a640 thread_name:rbd-mirror 2023-12-04T14:42:40.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:42:40.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f0842054db0] 2023-12-04T14:42:40.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f084209c39a] 2023-12-04T14:42:40.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 3: pthread_cond_wait() 2023-12-04T14:42:40.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f08424d5500] 2023-12-04T14:42:40.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 5: /lib64/librados.so.2(+0x13d02b) [0x7f0843cc602b] 2023-12-04T14:42:40.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x559a93527768] 2023-12-04T14:42:40.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x559a9351f7c8] 2023-12-04T14:42:40.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x559a934fee74] 2023-12-04T14:42:40.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 9: main() 2023-12-04T14:42:40.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f084203feb0] 2023-12-04T14:42:40.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 11: __libc_start_main() 2023-12-04T14:42:40.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 12: _start() 2023-12-04T14:42:40.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr:2023-12-04T14:42:40.484+0000 7f0841c1a640 -1 *** Caught signal (Aborted) ** 2023-12-04T14:42:40.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: in thread 7f0841c1a640 thread_name:rbd-mirror 2023-12-04T14:42:40.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 2023-12-04T14:42:40.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:42:40.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f0842054db0] 2023-12-04T14:42:40.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f084209c39a] 2023-12-04T14:42:40.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 3: pthread_cond_wait() 2023-12-04T14:42:40.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f08424d5500] 2023-12-04T14:42:40.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 5: /lib64/librados.so.2(+0x13d02b) [0x7f0843cc602b] 2023-12-04T14:42:40.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x559a93527768] 2023-12-04T14:42:40.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x559a9351f7c8] 2023-12-04T14:42:40.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x559a934fee74] 2023-12-04T14:42:40.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 9: main() 2023-12-04T14:42:40.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f084203feb0] 2023-12-04T14:42:40.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 11: __libc_start_main() 2023-12-04T14:42:40.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 12: _start() 2023-12-04T14:42:40.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T14:42:40.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 2023-12-04T14:42:40.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: -2744> 2023-12-04T14:32:35.340+0000 7f083ebbe640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 96361) UID: 1000 2023-12-04T14:42:40.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 0> 2023-12-04T14:42:40.484+0000 7f0841c1a640 -1 *** Caught signal (Aborted) ** 2023-12-04T14:42:40.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: in thread 7f0841c1a640 thread_name:rbd-mirror 2023-12-04T14:42:40.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 2023-12-04T14:42:40.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:42:40.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f0842054db0] 2023-12-04T14:42:40.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f084209c39a] 2023-12-04T14:42:40.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 3: pthread_cond_wait() 2023-12-04T14:42:40.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f08424d5500] 2023-12-04T14:42:40.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 5: /lib64/librados.so.2(+0x13d02b) [0x7f0843cc602b] 2023-12-04T14:42:40.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x559a93527768] 2023-12-04T14:42:40.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x559a9351f7c8] 2023-12-04T14:42:40.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x559a934fee74] 2023-12-04T14:42:40.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 9: main() 2023-12-04T14:42:40.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f084203feb0] 2023-12-04T14:42:40.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 11: __libc_start_main() 2023-12-04T14:42:40.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 12: _start() 2023-12-04T14:42:40.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T14:42:40.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 2023-12-04T14:42:40.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: -9999> 2023-12-04T14:32:35.340+0000 7f083ebbe640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 96361) UID: 1000 2023-12-04T14:42:40.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: -9998> 2023-12-04T14:42:40.484+0000 7f0841c1a640 -1 *** Caught signal (Aborted) ** 2023-12-04T14:42:40.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: in thread 7f0841c1a640 thread_name:rbd-mirror 2023-12-04T14:42:40.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 2023-12-04T14:42:40.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: ceph version 18.0.0-7672-g7c730b29 (7c730b2996d0b00d56aea5ca791f90c534a13422) reef (dev) 2023-12-04T14:42:40.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f0842054db0] 2023-12-04T14:42:40.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 2: /lib64/libc.so.6(+0x9c39a) [0x7f084209c39a] 2023-12-04T14:42:40.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 3: pthread_cond_wait() 2023-12-04T14:42:40.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f08424d5500] 2023-12-04T14:42:40.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 5: /lib64/librados.so.2(+0x13d02b) [0x7f0843cc602b] 2023-12-04T14:42:40.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x559a93527768] 2023-12-04T14:42:40.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x559a9351f7c8] 2023-12-04T14:42:40.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x559a934fee74] 2023-12-04T14:42:40.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 9: main() 2023-12-04T14:42:40.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f084203feb0] 2023-12-04T14:42:40.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 11: __libc_start_main() 2023-12-04T14:42:40.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 12: _start() 2023-12-04T14:42:40.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-04T14:42:40.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr: 2023-12-04T14:42:41.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi176.stderr:daemon-helper: command crashed with signal 6 2023-12-04T14:46:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:2023-12-04T14:46:13.188+0000 7fd195d7d640 0 monclient(hunting): authenticate timed out after 300 2023-12-04T14:46:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-04T14:46:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T14:46:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster2 service status 2023-12-04T14:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:2023-12-04T14:51:13.272+0000 7fccbd35a640 0 monclient(hunting): authenticate timed out after 300 2023-12-04T14:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-04T14:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:51:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:51:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-04T14:51:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-04T14:51:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror images 2023-12-04T14:51:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror images' 2023-12-04T14:51:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' ls -l 2023-12-04T14:56:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T14:56:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-04T14:56:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T14:56:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T14:56:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror mirror pool info 2023-12-04T14:56:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror mirror pool info' 2023-12-04T14:56:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' mirror pool info 2023-12-04T15:01:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T15:01:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T15:01:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T15:01:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror mirror pool status 2023-12-04T15:01:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror mirror pool status' 2023-12-04T15:01:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T15:01:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' mirror pool status --verbose 2023-12-04T15:06:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T15:06:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T15:06:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T15:06:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' ls 2023-12-04T15:11:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror rbd_mirroring omap vals 2023-12-04T15:11:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror rbd_mirroring omap vals' 2023-12-04T15:11:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirroring 2023-12-04T15:16:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-04T15:16:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror rbd_mirror_leader omap vals 2023-12-04T15:16:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror rbd_mirror_leader omap vals' 2023-12-04T15:16:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirror_leader 2023-12-04T15:21:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-04T15:21:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T15:21:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T15:21:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-04T15:21:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror ns1 images 2023-12-04T15:21:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror ns1 images' 2023-12-04T15:21:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 ls -l 2023-12-04T15:26:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T15:26:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-04T15:26:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T15:26:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T15:26:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirrorns1 mirror pool info 2023-12-04T15:26:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirrorns1 mirror pool info' 2023-12-04T15:26:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool info 2023-12-04T15:31:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T15:31:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T15:31:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirrorns1 mirror pool status 2023-12-04T15:31:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T15:31:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirrorns1 mirror pool status' 2023-12-04T15:31:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T15:31:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool status --verbose 2023-12-04T15:36:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T15:36:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T15:36:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T15:36:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 ls 2023-12-04T15:41:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror ns1 rbd_mirroring omap vals 2023-12-04T15:41:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror ns1 rbd_mirroring omap vals' 2023-12-04T15:41:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirroring 2023-12-04T15:46:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-04T15:46:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror ns1 rbd_mirror_leader omap vals 2023-12-04T15:46:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror ns1 rbd_mirror_leader omap vals' 2023-12-04T15:46:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2023-12-04T15:51:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-04T15:51:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T15:51:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T15:51:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-04T15:51:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror ns2 images 2023-12-04T15:51:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror ns2 images' 2023-12-04T15:51:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 ls -l 2023-12-04T15:56:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T15:56:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-04T15:56:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T15:56:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T15:56:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirrorns2 mirror pool info 2023-12-04T15:56:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirrorns2 mirror pool info' 2023-12-04T15:56:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool info 2023-12-04T16:01:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T16:01:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T16:01:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T16:01:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirrorns2 mirror pool status 2023-12-04T16:01:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirrorns2 mirror pool status' 2023-12-04T16:01:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T16:01:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool status --verbose 2023-12-04T16:06:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T16:06:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T16:06:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T16:06:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 ls 2023-12-04T16:11:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror ns2 rbd_mirroring omap vals 2023-12-04T16:11:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror ns2 rbd_mirroring omap vals' 2023-12-04T16:11:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirroring 2023-12-04T16:16:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-04T16:16:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror ns2 rbd_mirror_leader omap vals' 2023-12-04T16:16:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror ns2 rbd_mirror_leader omap vals 2023-12-04T16:16:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2023-12-04T16:21:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-04T16:21:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T16:21:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T16:21:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-04T16:21:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-04T16:21:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parent images 2023-12-04T16:21:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parent images' 2023-12-04T16:21:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' ls -l 2023-12-04T16:26:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T16:26:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-04T16:26:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T16:26:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T16:26:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parent mirror pool info 2023-12-04T16:26:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parent mirror pool info' 2023-12-04T16:26:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool info 2023-12-04T16:31:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T16:31:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T16:31:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T16:31:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parent mirror pool status 2023-12-04T16:31:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parent mirror pool status' 2023-12-04T16:31:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T16:31:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool status --verbose 2023-12-04T16:36:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T16:36:13.943 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T16:36:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T16:36:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' ls 2023-12-04T16:41:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parent rbd_mirroring omap vals 2023-12-04T16:41:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parent rbd_mirroring omap vals' 2023-12-04T16:41:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2023-12-04T16:46:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-04T16:46:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parent rbd_mirror_leader omap vals 2023-12-04T16:46:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parent rbd_mirror_leader omap vals' 2023-12-04T16:46:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2023-12-04T16:51:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-04T16:51:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T16:51:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T16:51:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-04T16:51:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parent ns1 images 2023-12-04T16:51:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parent ns1 images' 2023-12-04T16:51:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l 2023-12-04T16:56:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T16:56:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-04T16:56:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T16:56:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T16:56:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parentns1 mirror pool info' 2023-12-04T16:56:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parentns1 mirror pool info 2023-12-04T16:56:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool info 2023-12-04T17:01:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T17:01:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T17:01:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T17:01:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parentns1 mirror pool status 2023-12-04T17:01:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parentns1 mirror pool status' 2023-12-04T17:01:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T17:01:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool status --verbose 2023-12-04T17:06:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T17:06:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T17:06:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T17:06:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls 2023-12-04T17:11:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parent ns1 rbd_mirroring omap vals 2023-12-04T17:11:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parent ns1 rbd_mirroring omap vals' 2023-12-04T17:11:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2023-12-04T17:16:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-04T17:16:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parent ns1 rbd_mirror_leader omap vals 2023-12-04T17:16:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parent ns1 rbd_mirror_leader omap vals' 2023-12-04T17:16:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2023-12-04T17:21:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-04T17:21:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T17:21:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T17:21:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-12-04T17:21:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parent ns2 images 2023-12-04T17:21:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parent ns2 images' 2023-12-04T17:21:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls -l 2023-12-04T17:26:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T17:26:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-04T17:26:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T17:26:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T17:26:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parentns2 mirror pool info 2023-12-04T17:26:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parentns2 mirror pool info' 2023-12-04T17:26:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool info 2023-12-04T17:31:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T17:31:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T17:31:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T17:31:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parentns2 mirror pool status 2023-12-04T17:31:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parentns2 mirror pool status' 2023-12-04T17:31:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T17:31:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool status --verbose 2023-12-04T17:36:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:rbd: couldn't connect to the cluster! 2023-12-04T17:36:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T17:36:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T17:36:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls 2023-12-04T17:41:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parent ns2 rbd_mirroring omap vals 2023-12-04T17:41:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parent ns2 rbd_mirroring omap vals' 2023-12-04T17:41:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2023-12-04T17:46:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-04T17:46:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 mirror_parent ns2 rbd_mirror_leader omap vals 2023-12-04T17:46:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 mirror_parent ns2 rbd_mirror_leader omap vals' 2023-12-04T17:46:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2023-12-04T17:51:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-04T17:51:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 2023-12-04T17:51:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout: 2023-12-04T17:51:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local ret 2023-12-04T17:51:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-12-04T17:51:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 0 3 2023-12-04T17:51:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T17:51:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ daemon_pid_file cluster1:0 2023-12-04T17:51:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:0 2023-12-04T17:51:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T17:51:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-04T17:51:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:0 2023-12-04T17:51:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T17:51:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T17:51:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T17:51:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T17:51:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-04T17:51:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T17:51:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T17:51:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T17:51:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T17:51:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2023-12-04T17:51:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2023-12-04T17:51:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2023-12-04T17:51:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid ']' 2023-12-04T17:51:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist)' 2023-12-04T17:51:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist) 2023-12-04T17:51:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ continue 2023-12-04T17:51:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T17:51:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ daemon_pid_file cluster1:1 2023-12-04T17:51:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:1 2023-12-04T17:51:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T17:51:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-04T17:51:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:1 2023-12-04T17:51:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T17:51:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T17:51:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T17:51:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T17:51:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-04T17:51:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T17:51:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T17:51:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T17:51:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T17:51:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2023-12-04T17:51:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2023-12-04T17:51:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2023-12-04T17:51:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid ']' 2023-12-04T17:51:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist)' 2023-12-04T17:51:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist) 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ continue 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ daemon_pid_file cluster1:2 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:2 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:2 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T17:51:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T17:51:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T17:51:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2023-12-04T17:51:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2023-12-04T17:51:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2023-12-04T17:51:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid ']' 2023-12-04T17:51:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist)' 2023-12-04T17:51:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist) 2023-12-04T17:51:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ continue 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ daemon_pid_file cluster1:3 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster1:3 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster1:3 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster1 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster1 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster1 2023-12-04T17:51:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T17:51:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T17:51:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2023-12-04T17:51:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2023-12-04T17:51:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2023-12-04T17:51:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid ']' 2023-12-04T17:51:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist)' 2023-12-04T17:51:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist) 2023-12-04T17:51:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ continue 2023-12-04T17:51:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-12-04T17:51:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 0 3 2023-12-04T17:51:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ daemon_pid_file cluster2:0 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster2:0 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster2:0 cluster instance 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster2:0 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster2 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=0 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 0 = cluster2:0 ']' 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster2 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster2 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=0 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=0 2023-12-04T17:51:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2023-12-04T17:51:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2023-12-04T17:51:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2023-12-04T17:51:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid ']' 2023-12-04T17:51:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist)' 2023-12-04T17:51:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist) 2023-12-04T17:51:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ continue 2023-12-04T17:51:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T17:51:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ daemon_pid_file cluster2:1 2023-12-04T17:51:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster2:1 2023-12-04T17:51:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T17:51:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster2:1 cluster instance 2023-12-04T17:51:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster2:1 2023-12-04T17:51:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T17:51:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T17:51:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster2 2023-12-04T17:51:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=1 2023-12-04T17:51:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 1 = cluster2:1 ']' 2023-12-04T17:51:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster2 2023-12-04T17:51:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster2 2023-12-04T17:51:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=1 2023-12-04T17:51:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=1 2023-12-04T17:51:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2023-12-04T17:51:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2023-12-04T17:51:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2023-12-04T17:51:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid ']' 2023-12-04T17:51:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist)' 2023-12-04T17:51:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist) 2023-12-04T17:51:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ continue 2023-12-04T17:51:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T17:51:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ daemon_pid_file cluster2:2 2023-12-04T17:51:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster2:2 2023-12-04T17:51:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T17:51:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster2:2 cluster instance 2023-12-04T17:51:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster2:2 2023-12-04T17:51:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T17:51:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T17:51:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster2 2023-12-04T17:51:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=2 2023-12-04T17:51:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 2 = cluster2:2 ']' 2023-12-04T17:51:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster2 2023-12-04T17:51:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster2 2023-12-04T17:51:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=2 2023-12-04T17:51:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=2 2023-12-04T17:51:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2023-12-04T17:51:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2023-12-04T17:51:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2023-12-04T17:51:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid ']' 2023-12-04T17:51:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist)' 2023-12-04T17:51:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist) 2023-12-04T17:51:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ continue 2023-12-04T17:51:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T17:51:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ daemon_pid_file cluster2:3 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster=cluster2:3 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ set_cluster_instance cluster2:3 cluster instance 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local val=cluster2:3 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local cluster_var_name=cluster 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ local instance_var_name=instance 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ cluster=cluster2 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ instance=3 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ '[' 3 = cluster2:3 ']' 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval cluster=cluster2 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ cluster=cluster2 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ eval instance=3 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ instance=3 2023-12-04T17:51:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2023-12-04T17:51:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2023-12-04T17:51:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2023-12-04T17:51:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid ']' 2023-12-04T17:51:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist)' 2023-12-04T17:51:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist) 2023-12-04T17:51:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ continue 2023-12-04T17:51:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T17:51:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -z '' ']' 2023-12-04T17:51:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster instance 2023-12-04T17:51:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T17:51:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-04T17:56:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:2023-12-04T17:56:14.655+0000 7f71a20bb640 0 monclient(hunting): authenticate timed out after 300 2023-12-04T17:56:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-04T17:56:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T17:56:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-04T18:01:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:2023-12-04T18:01:14.742+0000 7fc1ac490640 0 monclient(hunting): authenticate timed out after 300 2023-12-04T18:01:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-04T18:01:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T18:01:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-04T18:06:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:2023-12-04T18:06:14.830+0000 7f4deb264640 0 monclient(hunting): authenticate timed out after 300 2023-12-04T18:06:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-04T18:06:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ CEPH_ARGS= 2023-12-04T18:06:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-04T18:11:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:2023-12-04T18:11:14.917+0000 7fe408578640 0 monclient(hunting): authenticate timed out after 300 2023-12-04T18:11:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-04T18:11:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-12-04T18:11:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stop_mirrors cluster1 2023-12-04T18:11:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1 2023-12-04T18:11:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local sig= 2023-12-04T18:11:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 0 3 2023-12-04T18:11:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T18:11:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stop_mirror cluster1:0 '' 2023-12-04T18:11:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1:0 2023-12-04T18:11:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local sig= 2023-12-04T18:11:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T18:11:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T18:11:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T18:11:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stop_mirror cluster1:1 '' 2023-12-04T18:11:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1:1 2023-12-04T18:11:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local sig= 2023-12-04T18:11:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T18:11:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T18:11:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T18:11:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stop_mirror cluster1:2 '' 2023-12-04T18:11:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1:2 2023-12-04T18:11:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local sig= 2023-12-04T18:11:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T18:11:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T18:11:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T18:11:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stop_mirror cluster1:3 '' 2023-12-04T18:11:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster1:3 2023-12-04T18:11:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local sig= 2023-12-04T18:11:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T18:11:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T18:11:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-12-04T18:11:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stop_mirrors cluster2 2023-12-04T18:11:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2 2023-12-04T18:11:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local sig= 2023-12-04T18:11:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:++ seq 0 3 2023-12-04T18:11:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T18:11:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stop_mirror cluster2:0 '' 2023-12-04T18:11:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2:0 2023-12-04T18:11:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local sig= 2023-12-04T18:11:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T18:11:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T18:11:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T18:11:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stop_mirror cluster2:1 '' 2023-12-04T18:11:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2:1 2023-12-04T18:11:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local sig= 2023-12-04T18:11:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T18:11:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T18:11:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T18:11:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stop_mirror cluster2:2 '' 2023-12-04T18:11:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2:2 2023-12-04T18:11:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local sig= 2023-12-04T18:11:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T18:11:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T18:11:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-04T18:11:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ stop_mirror cluster2:3 '' 2023-12-04T18:11:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local cluster=cluster2:3 2023-12-04T18:11:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ local sig= 2023-12-04T18:11:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test -n 1 2023-12-04T18:11:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ return 2023-12-04T18:11:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' -z 1 ']' 2023-12-04T18:11:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ test '' = /tmp/tmp.rDwoQpiP9p 2023-12-04T18:11:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ rm -Rf /tmp/tmp.rDwoQpiP9p 2023-12-04T18:11:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ '[' 110 -eq 0 ']' 2023-12-04T18:11:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ echo FAIL 2023-12-04T18:11:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stdout:FAIL 2023-12-04T18:11:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-12-04T18:11:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi176.stderr:+ exit 110 2023-12-04T18:11:14.936 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2023-12-04T18:11:14.936 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2023-12-04T18:11:15.212 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi176 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c730b2996d0b00d56aea5ca791f90c534a13422 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2023-12-04T18:11:15.677 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=aa28831dc0f54a009e0a1fb4516b61fb Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi176 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c730b2996d0b00d56aea5ca791f90c534a13422 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2023-12-04T18:11:15.679 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-04T18:11:15.690 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-04T18:11:15.690 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-12-04T18:11:15.730 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-04T18:11:15.740 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-04T18:11:15.740 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 211, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-12-04T18:11:15.744 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-04T18:11:15.754 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2023-12-04T18:11:15.755 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:11:15.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi176.stderr:2023-12-04T18:11:15.754+0000 7f3c3e820640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 111224) UID: 1000 2023-12-04T18:16:14.813 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-04T18:16:14.839 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-04T18:16:14.850 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2023-12-04T18:16:14.850 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:16:14.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi176.stderr:2023-12-04T18:16:14.850+0000 7f615358f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 107448) UID: 1000 2023-12-04T18:21:13.923 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-04T18:21:13.929 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-04T18:21:13.941 ERROR:tasks.rbd_mirror.cluster2.client.mirror.1:tried to stop a non-running daemon 2023-12-04T18:21:13.945 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-04T18:21:13.956 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2023-12-04T18:21:13.957 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:21:13.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi176.stderr:2023-12-04T18:21:13.956+0000 7f050dbf2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 104781) UID: 1000 2023-12-04T18:26:13.029 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-04T18:26:13.034 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-04T18:26:13.048 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2023-12-04T18:26:13.048 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:26:13.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi176.stderr:2023-12-04T18:26:13.048+0000 7faa48576640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 91845) UID: 1000 2023-12-04T18:31:12.118 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-04T18:31:12.123 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-04T18:31:12.135 ERROR:tasks.rbd_mirror.cluster1.client.mirror.2:tried to stop a non-running daemon 2023-12-04T18:31:12.138 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-04T18:31:12.152 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2023-12-04T18:31:12.152 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:31:12.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi176.stderr:2023-12-04T18:31:12.151+0000 7f78de776640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 105853) UID: 1000 2023-12-04T18:36:11.218 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-04T18:36:11.226 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-04T18:36:58.018 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2023-12-04T18:36:58.018 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:36:58.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi176.stderr:2023-12-04T18:36:58.018+0000 7fc110b23640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 112373) UID: 1000 2023-12-04T18:41:57.083 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-12-04T18:41:57.088 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-04T18:41:57.102 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-04T18:41:57.102 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-04T18:43:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-04T18:43:57.143 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi176 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-04T18:43:57.198 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-04T18:43:57.198 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-04T18:43:57.199 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2023-12-04T18:43:57.199 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:43:57.202 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-04T18:43:57.202 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi176 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi176 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2023-12-04T18:43:57.213 INFO:tasks.ceph.cluster2.osd.0:Stopped 2023-12-04T18:43:57.213 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2023-12-04T18:43:57.213 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:43:57.214 INFO:tasks.ceph.cluster2.osd.1:Stopped 2023-12-04T18:43:57.214 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2023-12-04T18:43:57.214 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:43:57.214 INFO:tasks.ceph.cluster2.osd.2:Stopped 2023-12-04T18:43:57.214 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-04T18:43:57.214 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2023-12-04T18:43:57.214 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:43:57.215 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2023-12-04T18:43:57.215 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-04T18:43:57.215 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2023-12-04T18:43:57.215 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:43:57.215 INFO:tasks.ceph.cluster2.mon.a:Stopped 2023-12-04T18:43:57.215 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-04T18:43:57.216 DEBUG:teuthology.orchestra.run.smithi176:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-04T18:43:57.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi176.front.sepia.ceph.com 2023-12-04T18:43:57.330 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2023-12-04T18:43:57.562 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi176.front.sepia.ceph.com 2023-12-04T18:43:57.562 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2023-12-04T18:43:57.726 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi176.front.sepia.ceph.com 2023-12-04T18:43:57.726 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2023-12-04T18:43:57.874 INFO:tasks.ceph:Archiving mon data... 2023-12-04T18:43:57.876 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148/data/cluster2.mon.a.tgz 2023-12-04T18:43:57.876 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2023-12-04T18:43:57.890 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.jqnhNYip4i 2023-12-04T18:43:57.890 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.jqnhNYip4i 2023-12-04T18:43:58.059 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.jqnhNYip4i 2023-12-04T18:43:58.143 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.jqnhNYip4i is 441KB 2023-12-04T18:43:58.211 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.jqnhNYip4i 2023-12-04T18:43:58.225 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-04T18:43:58.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-04T18:43:58.229 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-04T18:43:58.291 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi176 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-04T18:43:58.302 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-04T18:43:58.314 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-04T18:43:58.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-04T18:45:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-04T18:45:58.354 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2023-12-04T18:45:58.369 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-04T18:45:58.369 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-04T18:45:58.369 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2023-12-04T18:45:58.369 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:45:58.373 INFO:tasks.ceph.cluster1.osd.0:Stopped 2023-12-04T18:45:58.373 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2023-12-04T18:45:58.373 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:45:58.373 INFO:tasks.ceph.cluster1.osd.1:Stopped 2023-12-04T18:45:58.373 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2023-12-04T18:45:58.373 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:45:58.374 INFO:tasks.ceph.cluster1.osd.2:Stopped 2023-12-04T18:45:58.374 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-04T18:45:58.374 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2023-12-04T18:45:58.374 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:45:58.374 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2023-12-04T18:45:58.374 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-04T18:45:58.375 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2023-12-04T18:45:58.375 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T18:45:58.375 INFO:tasks.ceph.cluster1.mon.a:Stopped 2023-12-04T18:45:58.375 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-04T18:45:58.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-04T18:45:58.473 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi146.front.sepia.ceph.com 2023-12-04T18:45:58.473 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2023-12-04T18:45:58.538 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi146.front.sepia.ceph.com 2023-12-04T18:45:58.538 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2023-12-04T18:45:58.602 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi146.front.sepia.ceph.com 2023-12-04T18:45:58.602 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2023-12-04T18:45:58.669 INFO:tasks.ceph:Archiving mon data... 2023-12-04T18:45:58.670 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148/data/cluster1.mon.a.tgz 2023-12-04T18:45:58.670 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2023-12-04T18:45:58.685 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.p9vbEjAdLo 2023-12-04T18:45:58.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.p9vbEjAdLo 2023-12-04T18:45:58.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.p9vbEjAdLo 2023-12-04T18:45:58.910 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.p9vbEjAdLo is 290KB 2023-12-04T18:45:58.976 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.p9vbEjAdLo 2023-12-04T18:45:58.990 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-04T18:45:58.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-04T18:45:59.032 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-04T18:45:59.111 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-04T18:45:59.128 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-04T18:45:59.128 INFO:tasks.ceph:Archiving crash dumps... 2023-12-04T18:45:59.130 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148/remote/smithi146/crash 2023-12-04T18:45:59.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-04T18:45:59.178 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148/remote/smithi176/crash 2023-12-04T18:45:59.179 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-04T18:45:59.254 INFO:tasks.ceph:Compressing logs... 2023-12-04T18:45:59.255 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-04T18:45:59.257 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-04T18:45:59.280 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2023-12-04T18:45:59.296 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2023-12-04T18:45:59.296 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90112.log 2023-12-04T18:45:59.296 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89860.log 2023-12-04T18:45:59.296 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.90112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89902.log 2023-12-04T18:45:59.297 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90112.log.gz 2023-12-04T18:45:59.297 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.89860.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2023-12-04T18:45:59.297 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89860.log.gz 2023-12-04T18:45:59.297 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.89902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90364.log 2023-12-04T18:45:59.297 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89902.log.gz 2023-12-04T18:45:59.297 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2023-12-04T18:45:59.297 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.90364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85179.log 2023-12-04T18:45:59.297 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90364.log.gz 2023-12-04T18:45:59.297 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90026.log 2023-12-04T18:45:59.298 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.85179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84941.log 2023-12-04T18:45:59.298 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85179.log.gz 2023-12-04T18:45:59.298 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.90026.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2023-12-04T18:45:59.298 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90026.log.gz 2023-12-04T18:45:59.298 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.84941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85278.log 2023-12-04T18:45:59.298 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.84941.log.gz 2023-12-04T18:45:59.298 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90298.log 2023-12-04T18:45:59.298 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.85278.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85278.log.gz 2023-12-04T18:45:59.299 INFO:teuthology.orchestra.run.smithi146.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.90448.log 2023-12-04T18:45:59.299 INFO:teuthology.orchestra.run.smithi146.stderr: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2023-12-04T18:45:59.299 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.90298.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90298.log.gz 2023-12-04T18:45:59.299 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2023-12-04T18:45:59.299 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.90448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90024.log 2023-12-04T18:45:59.299 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90448.log.gz 2023-12-04T18:45:59.299 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90406.log 2023-12-04T18:45:59.299 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.90024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90238.log 2023-12-04T18:45:59.299 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90024.log.gz 2023-12-04T18:45:59.299 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.90406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89818.log 2023-12-04T18:45:59.300 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90406.log.gz 2023-12-04T18:45:59.300 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.90238.log: -- /var/log/ceph/cluster1-client.admin.85139.log 2023-12-04T18:45:59.300 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90238.log.gz 2023-12-04T18:45:59.300 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.89818.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89818.log.gz 2023-12-04T18:45:59.300 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85346.log 2023-12-04T18:45:59.300 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5/var/log/ceph/cluster1-client.admin.85139.log: --verbose -- /var/log/ceph/cluster1-osd.0.log 2023-12-04T18:45:59.300 INFO:teuthology.orchestra.run.smithi146.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85139.log.gz 2023-12-04T18:45:59.300 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.85346.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85346.log.gz 2023-12-04T18:45:59.300 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.89720.log 2023-12-04T18:45:59.301 INFO:teuthology.orchestra.run.smithi146.stderr: 88.9% -- replaced with /var/log/ceph/cluster1.log.gz 2023-12-04T18:45:59.301 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89568.log 2023-12-04T18:45:59.301 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-osd.0.log: /var/log/ceph/cluster1-client.admin.89720.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89720.log.gz 2023-12-04T18:45:59.301 INFO:teuthology.orchestra.run.smithi146.stderr: -- /var/log/ceph/cluster1-client.admin.90258.log 2023-12-04T18:45:59.301 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.89568.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89568.log.gz 2023-12-04T18:45:59.301 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85099.log 2023-12-04T18:45:59.301 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.90258.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90258.log.gz 2023-12-04T18:45:59.301 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80428.log 2023-12-04T18:45:59.301 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.85099.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85099.log.gz 2023-12-04T18:45:59.302 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89776.log 2023-12-04T18:45:59.302 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.tmp-client.admin.80428.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80428.log.gz 2023-12-04T18:45:59.302 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90152.log 2023-12-04T18:45:59.302 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.89776.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89776.log.gz 2023-12-04T18:45:59.302 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92482.log 2023-12-04T18:45:59.302 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.90152.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90152.log.gz 2023-12-04T18:45:59.302 INFO:teuthology.orchestra.run.smithi146.stderr: -- /var/log/ceph/cluster1-mon.a.log 2023-12-04T18:45:59.302 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.92482.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92482.log.gz 2023-12-04T18:45:59.303 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90025.log 2023-12-04T18:45:59.303 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89944.log 2023-12-04T18:45:59.303 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.90025.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90151.log 2023-12-04T18:45:59.303 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.90025.log.gz 2023-12-04T18:45:59.303 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.89944.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89944.log.gz 2023-12-04T18:45:59.303 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cluster1-client.admin.90151.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90151.log.gz 2023-12-04T18:45:59.307 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97844.log 2023-12-04T18:45:59.308 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115805.log 2023-12-04T18:45:59.309 INFO:teuthology.orchestra.run.smithi176.stderr:gzip/var/log/ceph/cluster1-client.mirror.97844.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115565.log 2023-12-04T18:45:59.309 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97084.log 2023-12-04T18:45:59.309 INFO:teuthology.orchestra.run.smithi176.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97844.log.gz 2023-12-04T18:45:59.310 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115565.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115805.log.gz 2023-12-04T18:45:59.310 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109073.log 2023-12-04T18:45:59.310 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106537.log 2023-12-04T18:45:59.310 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115565.log.gz 2023-12-04T18:45:59.310 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.97084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.91300.log 2023-12-04T18:45:59.310 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109258.log 2023-12-04T18:45:59.331 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101509.log 2023-12-04T18:45:59.332 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109073.log: /var/log/ceph/cluster2-client.mirror.109258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107651.log 2023-12-04T18:45:59.332 INFO:teuthology.orchestra.run.smithi176.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97084.log.gz 2023-12-04T18:45:59.332 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106537.log: /var/log/ceph/cluster2-client.mirror.3.91300.log: gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.101509.log: /var/log/ceph/cluster2-client.mirror.92182.log 2023-12-04T18:45:59.332 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109073.log.gz 2023-12-04T18:45:59.332 INFO:teuthology.orchestra.run.smithi176.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101509.log.gz 2023-12-04T18:45:59.332 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107651.log: 89.0% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.106537.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.109258.log.gz 2023-12-04T18:45:59.332 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.332 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118219.log 2023-12-04T18:45:59.332 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94107.log 2023-12-04T18:45:59.333 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.92182.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107651.log.gz 2023-12-04T18:45:59.333 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.118219.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98036.log 2023-12-04T18:45:59.333 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92182.log.gz 2023-12-04T18:45:59.333 INFO:teuthology.orchestra.run.smithi176.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.118219.log.gz 2023-12-04T18:45:59.333 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118239.log 2023-12-04T18:45:59.333 INFO:teuthology.orchestra.run.smithi176.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.94107.log.gz 2023-12-04T18:45:59.333 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98036.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110933.log 2023-12-04T18:45:59.333 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115745.log 2023-12-04T18:45:59.333 INFO:teuthology.orchestra.run.smithi176.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98036.log.gz 2023-12-04T18:45:59.334 INFO:teuthology.orchestra.run.smithi176.stderr: 10.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118239.log.gz 2023-12-04T18:45:59.334 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117550.log 2023-12-04T18:45:59.334 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118015.log 2023-12-04T18:45:59.334 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115745.log.gz 2023-12-04T18:45:59.334 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101368.log 2023-12-04T18:45:59.334 INFO:teuthology.orchestra.run.smithi176.stderr: 67.9% -- replaced with /var/log/ceph/cluster1-client.mirror.117550.log.gz 2023-12-04T18:45:59.334 INFO:teuthology.orchestra.run.smithi176.stderr:gzip/var/log/ceph/cluster2-client.admin.118015.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99540.log 2023-12-04T18:45:59.335 INFO:teuthology.orchestra.run.smithi176.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.118015.log.gz 2023-12-04T18:45:59.335 INFO:teuthology.orchestra.run.smithi176.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110933.log.gz 2023-12-04T18:45:59.335 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101368.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104333.log 2023-12-04T18:45:59.335 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.99540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.101842.log 2023-12-04T18:45:59.335 INFO:teuthology.orchestra.run.smithi176.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101368.log.gz 2023-12-04T18:45:59.335 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99540.log.gz 2023-12-04T18:45:59.335 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109832.log 2023-12-04T18:45:59.335 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.1.101842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95619.log 2023-12-04T18:45:59.336 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114545.log 2023-12-04T18:45:59.336 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109832.log.gz 2023-12-04T18:45:59.336 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118333.log 2023-12-04T18:45:59.336 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116805.log 2023-12-04T18:45:59.336 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114545.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.95619.log.gz 2023-12-04T18:45:59.336 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115165.log 2023-12-04T18:45:59.336 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114545.log.gz 2023-12-04T18:45:59.336 INFO:teuthology.orchestra.run.smithi176.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118333.log.gz 2023-12-04T18:45:59.336 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99588.log 2023-12-04T18:45:59.336 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116805.log.gz 2023-12-04T18:45:59.337 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106858.log 2023-12-04T18:45:59.337 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109025.log 2023-12-04T18:45:59.337 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99588.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115165.log.gz 2023-12-04T18:45:59.337 INFO:teuthology.orchestra.run.smithi176.stderr: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99588.log.gz 2023-12-04T18:45:59.337 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116945.log 2023-12-04T18:45:59.337 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109025.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106858.log.gz 2023-12-04T18:45:59.337 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95293.log 2023-12-04T18:45:59.337 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109025.log.gz 2023-12-04T18:45:59.337 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93654.log 2023-12-04T18:45:59.338 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116945.log.gz 2023-12-04T18:45:59.338 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99108.log 2023-12-04T18:45:59.338 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98667.log 2023-12-04T18:45:59.338 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.99108.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.93654.log.gz 2023-12-04T18:45:59.338 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.101842.log.gz 2023-12-04T18:45:59.338 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102959.log 2023-12-04T18:45:59.338 INFO:teuthology.orchestra.run.smithi176.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99108.log.gz 2023-12-04T18:45:59.338 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110017.log 2023-12-04T18:45:59.339 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98667.log: /var/log/ceph/cluster2-client.mirror.102959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118321.log 2023-12-04T18:45:59.339 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110017.log: 87.4%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.102959.log.gz 2023-12-04T18:45:59.339 INFO:teuthology.orchestra.run.smithi176.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.96378.log 2023-12-04T18:45:59.339 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118321.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118321.log.gz 2023-12-04T18:45:59.339 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93466.log 2023-12-04T18:45:59.339 INFO:teuthology.orchestra.run.smithi176.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.110017.log.gz 2023-12-04T18:45:59.339 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.1.96378.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111818.log 2023-12-04T18:45:59.339 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93466.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102651.log 2023-12-04T18:45:59.339 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111818.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.93466.log.gz 2023-12-04T18:45:59.340 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.103683.log 2023-12-04T18:45:59.340 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111818.log.gz 2023-12-04T18:45:59.340 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117421.log 2023-12-04T18:45:59.340 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102651.log: /var/log/ceph/cluster2-client.mirror.0.103683.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116505.log 2023-12-04T18:45:59.340 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95293.log.gz 2023-12-04T18:45:59.340 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.117421.log: 71.3% 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.3.91300.log.gz 2023-12-04T18:45:59.340 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93242.log 2023-12-04T18:45:59.340 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.117421.log.gz 2023-12-04T18:45:59.340 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118477.log 2023-12-04T18:45:59.341 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93242.log: gzip -5 --verbose 87.3% -- /var/log/ceph/cluster2-client.mirror.101251.log -- replaced with /var/log/ceph/cluster1-client.mirror.116505.log.gz 2023-12-04T18:45:59.341 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.341 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.118477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99397.log 2023-12-04T18:45:59.341 INFO:teuthology.orchestra.run.smithi176.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.118477.log.gz 2023-12-04T18:45:59.341 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101251.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98667.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115005.log 2023-12-04T18:45:59.341 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.341 INFO:teuthology.orchestra.run.smithi176.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.103683.log.gz 2023-12-04T18:45:59.341 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101467.log 2023-12-04T18:45:59.342 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115005.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99397.log.gz 2023-12-04T18:45:59.342 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114645.log 2023-12-04T18:45:59.342 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115005.log.gz 2023-12-04T18:45:59.342 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97868.log 2023-12-04T18:45:59.342 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101467.log: /var/log/ceph/cluster1-client.mirror.114645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112708.log 2023-12-04T18:45:59.342 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.97868.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101467.log.gz 2023-12-04T18:45:59.342 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114645.log.gz 2023-12-04T18:45:59.342 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117661.log 2023-12-04T18:45:59.342 INFO:teuthology.orchestra.run.smithi176.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97868.log.gz 2023-12-04T18:45:59.343 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.112708.log: --verbose -- /var/log/ceph/cluster1-client.mirror.114465.log 2023-12-04T18:45:59.343 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117661.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112708.log.gz 10.2% -- replaced with /var/log/ceph/cluster1-client.mirror.117661.log.gz 2023-12-04T18:45:59.343 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.343 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.100888.log 2023-12-04T18:45:59.343 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.114465.log: -- /var/log/ceph/cluster2-client.mirror.99930.log 2023-12-04T18:45:59.343 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.2.100888.log: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90916.log 2023-12-04T18:45:59.343 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.114465.log.gz 2023-12-04T18:45:59.343 INFO:teuthology.orchestra.run.smithi176.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93242.log.gz 2023-12-04T18:45:59.343 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94597.log 2023-12-04T18:45:59.343 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113696.log 2023-12-04T18:45:59.344 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90916.log.gz 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99930.log.gz 2023-12-04T18:45:59.344 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.344 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102355.log 2023-12-04T18:45:59.344 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98060.log 2023-12-04T18:45:59.344 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113696.log: /var/log/ceph/cluster2-client.mirror.102355.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113696.log.gz 2023-12-04T18:45:59.344 INFO:teuthology.orchestra.run.smithi176.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102355.log.gz 2023-12-04T18:45:59.344 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94480.log 2023-12-04T18:45:59.344 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97276.log 2023-12-04T18:45:59.359 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98060.log: /var/log/ceph/cluster1-client.mirror.94480.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98060.log.gz 2023-12-04T18:45:59.359 INFO:teuthology.orchestra.run.smithi176.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94480.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94660.log 2023-12-04T18:45:59.359 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.359 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113371.log 2023-12-04T18:45:59.359 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94660.log: /var/log/ceph/cluster1-client.mirror.97276.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97276.log.gz 2023-12-04T18:45:59.359 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91623.log 2023-12-04T18:45:59.360 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113371.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113371.log.gz 2023-12-04T18:45:59.360 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102767.log 2023-12-04T18:45:59.360 INFO:teuthology.orchestra.run.smithi176.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.94597.log.gz 2023-12-04T18:45:59.360 INFO:teuthology.orchestra.run.smithi176.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.100888.log.gz 2023-12-04T18:45:59.360 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115225.log 2023-12-04T18:45:59.360 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.91623.log: /var/log/ceph/cluster2-client.mirror.102767.log: 67.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89782.log 2023-12-04T18:45:59.360 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.91623.log.gz/var/log/ceph/cluster1-client.mirror.115225.log: 2023-12-04T18:45:59.360 INFO:teuthology.orchestra.run.smithi176.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102767.log.gz 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115225.log.gz 2023-12-04T18:45:59.360 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99237.log 2023-12-04T18:45:59.361 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.361 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101330.log 2023-12-04T18:45:59.361 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.89782.log: /var/log/ceph/cluster2-client.mirror.99237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115665.log 2023-12-04T18:45:59.361 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89782.log.gz 2023-12-04T18:45:59.361 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101330.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91000.log 2023-12-04T18:45:59.361 INFO:teuthology.orchestra.run.smithi176.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.99237.log.gz 2023-12-04T18:45:59.361 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115665.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105140.log 2023-12-04T18:45:59.361 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.91000.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91000.log.gz 2023-12-04T18:45:59.362 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115665.log.gz 2023-12-04T18:45:59.362 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.94660.log.gz 2023-12-04T18:45:59.362 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108290.log 2023-12-04T18:45:59.362 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.105140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111798.log 2023-12-04T18:45:59.362 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108290.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105140.log.gz 2023-12-04T18:45:59.362 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112301.log 2023-12-04T18:45:59.362 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.111798.log: -- replaced with /var/log/ceph/cluster2-client.mirror.108290.log.gz 2023-12-04T18:45:59.362 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112544.log 2023-12-04T18:45:59.363 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112301.log: gzip 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111798.log.gz 2023-12-04T18:45:59.363 INFO:teuthology.orchestra.run.smithi176.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114328.log 2023-12-04T18:45:59.363 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112301.log.gz 2023-12-04T18:45:59.363 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.112544.log: -- /var/log/ceph/cluster1-client.mirror.115245.log 2023-12-04T18:45:59.363 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104254.log 2023-12-04T18:45:59.363 INFO:teuthology.orchestra.run.smithi176.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114328.log.gz 2023-12-04T18:45:59.363 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111866.log 2023-12-04T18:45:59.363 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104254.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115245.log.gz 2023-12-04T18:45:59.364 INFO:teuthology.orchestra.run.smithi176.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112544.log.gz 2023-12-04T18:45:59.364 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97456.log 2023-12-04T18:45:59.364 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111142.log 2023-12-04T18:45:59.364 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111866.log: /var/log/ceph/cluster1-client.mirror.97456.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111866.log.gz 2023-12-04T18:45:59.364 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93606.log 2023-12-04T18:45:59.364 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111142.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.111142.log.gz 2023-12-04T18:45:59.364 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107699.log 2023-12-04T18:45:59.364 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93606.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.93606.log.gz 2023-12-04T18:45:59.364 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97932.log 2023-12-04T18:45:59.364 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107699.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107699.log.gz 2023-12-04T18:45:59.365 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103159.log 2023-12-04T18:45:59.365 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.97932.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97456.log.gz 2023-12-04T18:45:59.365 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102981.log 2023-12-04T18:45:59.365 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103159.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103159.log.gz 2023-12-04T18:45:59.365 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109461.log 2023-12-04T18:45:59.365 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102981.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97932.log.gz 2023-12-04T18:45:59.365 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98647.log 2023-12-04T18:45:59.365 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109461.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109461.log.gz 2023-12-04T18:45:59.365 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103480.log 2023-12-04T18:45:59.367 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98647.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98647.log.gz 2023-12-04T18:45:59.367 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103539.log 2023-12-04T18:45:59.367 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103480.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103480.log.gz 2023-12-04T18:45:59.367 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94243.log 2023-12-04T18:45:59.379 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117737.log 2023-12-04T18:45:59.384 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94243.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94243.log.gz 2023-12-04T18:45:59.386 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109627.log 2023-12-04T18:45:59.399 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117737.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.117737.log.gz 2023-12-04T18:45:59.402 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90790.log 2023-12-04T18:45:59.403 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109627.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.109627.log.gz 2023-12-04T18:45:59.410 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110386.log 2023-12-04T18:45:59.411 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.90790.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90790.log.gz 2023-12-04T18:45:59.419 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105535.log 2023-12-04T18:45:59.430 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90032.log 2023-12-04T18:45:59.430 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105535.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105535.log.gz 2023-12-04T18:45:59.432 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110386.log.gz 2023-12-04T18:45:59.438 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103139.log 2023-12-04T18:45:59.451 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90032.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90032.log.gz 2023-12-04T18:45:59.454 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118079.log 2023-12-04T18:45:59.471 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103139.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102651.log.gz 2023-12-04T18:45:59.475 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115825.log 2023-12-04T18:45:59.475 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.118079.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.118079.log.gz 2023-12-04T18:45:59.488 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109692.log 2023-12-04T18:45:59.489 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115825.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115825.log.gz 2023-12-04T18:45:59.502 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117883.log 2023-12-04T18:45:59.503 INFO:teuthology.orchestra.run.smithi146.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2023-12-04T18:45:59.504 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109692.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109692.log.gz 2023-12-04T18:45:59.518 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116465.log 2023-12-04T18:45:59.518 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117883.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.117883.log.gz 2023-12-04T18:45:59.526 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99031.log 2023-12-04T18:45:59.531 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116465.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116465.log.gz 2023-12-04T18:45:59.543 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94083.log 2023-12-04T18:45:59.544 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99031.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99031.log.gz 2023-12-04T18:45:59.547 INFO:teuthology.orchestra.run.smithi146.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2023-12-04T18:45:59.548 INFO:teuthology.orchestra.run.smithi176.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.96378.log.gz 2023-12-04T18:45:59.549 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95137.log 2023-12-04T18:45:59.550 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94083.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.94083.log.gz 2023-12-04T18:45:59.551 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116285.log 2023-12-04T18:45:59.551 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95137.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.95137.log.gz 2023-12-04T18:45:59.552 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114106.log 2023-12-04T18:45:59.553 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116285.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116285.log.gz 2023-12-04T18:45:59.553 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116765.log 2023-12-04T18:45:59.555 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114106.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114106.log.gz 2023-12-04T18:45:59.555 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93149.log 2023-12-04T18:45:59.556 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116765.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116765.log.gz 2023-12-04T18:45:59.556 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118466.log 2023-12-04T18:45:59.557 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93149.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93149.log.gz 2023-12-04T18:45:59.557 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114505.log 2023-12-04T18:45:59.558 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.118466.log: 7.1% -- replaced with /var/log/ceph/cluster1-client.admin.118466.log.gz 2023-12-04T18:45:59.559 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91948.log 2023-12-04T18:45:59.560 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114505.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114505.log.gz 2023-12-04T18:45:59.560 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101410.log 2023-12-04T18:45:59.560 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.91948.log: 66.9% -- replaced with /var/log/ceph/cluster2-client.admin.91948.log.gz 2023-12-04T18:45:59.561 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95315.log 2023-12-04T18:45:59.576 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100282.log 2023-12-04T18:45:59.577 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95315.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103539.log.gz 2023-12-04T18:45:59.577 INFO:teuthology.orchestra.run.smithi176.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95315.log.gz 2023-12-04T18:45:59.577 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106212.log 2023-12-04T18:45:59.578 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100282.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100282.log.gz 2023-12-04T18:45:59.578 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107603.log 2023-12-04T18:45:59.580 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106212.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.106212.log.gz 2023-12-04T18:45:59.580 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105740.log 2023-12-04T18:45:59.581 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107603.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107603.log.gz 2023-12-04T18:45:59.581 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95250.log 2023-12-04T18:45:59.583 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105740.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105740.log.gz 2023-12-04T18:45:59.583 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106465.log 2023-12-04T18:45:59.588 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95250.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95250.log.gz 2023-12-04T18:45:59.589 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96378.log 2023-12-04T18:45:59.590 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106465.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.106465.log.gz 2023-12-04T18:45:59.591 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112732.log 2023-12-04T18:45:59.600 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96378.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101330.log.gz 2023-12-04T18:45:59.602 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112889.log 2023-12-04T18:45:59.602 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112732.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112732.log.gz 2023-12-04T18:45:59.606 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90832.log 2023-12-04T18:45:59.606 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112889.log: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.96378.log.gz 2023-12-04T18:45:59.606 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112889.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97252.log 2023-12-04T18:45:59.606 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.606 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117831.log 2023-12-04T18:45:59.606 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90832.log: /var/log/ceph/cluster2-client.mirror.97252.log: gzip 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90832.log.gz 2023-12-04T18:45:59.606 INFO:teuthology.orchestra.run.smithi176.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96982.log 2023-12-04T18:45:59.607 INFO:teuthology.orchestra.run.smithi176.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.97252.log.gz 2023-12-04T18:45:59.607 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117831.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.117831.log.gz 2023-12-04T18:45:59.607 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109208.log 2023-12-04T18:45:59.607 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.96982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116525.log 2023-12-04T18:45:59.607 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109208.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.96982.log.gz 2023-12-04T18:45:59.612 INFO:teuthology.orchestra.run.smithi176.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92123.log 2023-12-04T18:45:59.613 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116525.log: -- replaced with /var/log/ceph/cluster2-client.mirror.109208.log.gz 2023-12-04T18:45:59.613 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104153.log 2023-12-04T18:45:59.613 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116525.log.gz 2023-12-04T18:45:59.613 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.92123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115365.log 2023-12-04T18:45:59.613 INFO:teuthology.orchestra.run.smithi176.stderr: 54.5%/var/log/ceph/cluster2-client.mirror.104153.log: -- replaced with /var/log/ceph/cluster2-client.mirror.92123.log.gz 2023-12-04T18:45:59.613 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104153.log.gz 2023-12-04T18:45:59.613 INFO:teuthology.orchestra.run.smithi176.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.107991.log 2023-12-04T18:45:59.614 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106927.log 2023-12-04T18:45:59.614 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115365.log: /var/log/ceph/cluster2-client.mirror.107991.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115365.log.gz 2023-12-04T18:45:59.614 INFO:teuthology.orchestra.run.smithi176.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107991.log.gz 2023-12-04T18:45:59.614 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116145.log 2023-12-04T18:45:59.614 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108386.log 2023-12-04T18:45:59.614 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116145.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116145.log.gz 2023-12-04T18:45:59.614 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.104798.log 2023-12-04T18:45:59.614 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108386.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106927.log.gz 2023-12-04T18:45:59.615 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.108386.log.gz 2023-12-04T18:45:59.615 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102572.log 2023-12-04T18:45:59.615 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.0.104798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103683.log 2023-12-04T18:45:59.626 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117498.log 2023-12-04T18:45:59.626 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.103683.log: 80.7% -- replaced with /var/log/ceph/cluster1-client.mirror.103683.log.gz 2023-12-04T18:45:59.640 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108892.log 2023-12-04T18:45:59.641 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117498.log: 68.1% -- replaced with /var/log/ceph/cluster1-client.mirror.117498.log.gz 2023-12-04T18:45:59.648 INFO:teuthology.orchestra.run.smithi176.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.104798.log.gz 2023-12-04T18:45:59.649 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105234.log 2023-12-04T18:45:59.649 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108892.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108892.log.gz 2023-12-04T18:45:59.650 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115025.log 2023-12-04T18:45:59.650 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105234.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105234.log.gz 2023-12-04T18:45:59.651 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112913.log 2023-12-04T18:45:59.652 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115025.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115025.log.gz 2023-12-04T18:45:59.652 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104911.log 2023-12-04T18:45:59.654 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112913.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112913.log.gz 2023-12-04T18:45:59.654 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101036.log 2023-12-04T18:45:59.654 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104911.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.104911.log.gz 2023-12-04T18:45:59.656 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111935.log 2023-12-04T18:45:59.656 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101036.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101036.log.gz 2023-12-04T18:45:59.656 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104175.log 2023-12-04T18:45:59.658 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111935.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111935.log.gz 2023-12-04T18:45:59.658 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116605.log 2023-12-04T18:45:59.670 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102493.log 2023-12-04T18:45:59.670 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116605.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116605.log.gz 2023-12-04T18:45:59.683 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92785.log 2023-12-04T18:45:59.694 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102493.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103139.log.gz 2023-12-04T18:45:59.701 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98217.log 2023-12-04T18:45:59.721 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.92785.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98130.log 2023-12-04T18:45:59.726 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98217.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98217.log.gz 2023-12-04T18:45:59.736 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104333.log.gz 2023-12-04T18:45:59.736 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117626.log 2023-12-04T18:45:59.741 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98130.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.98130.log.gz 2023-12-04T18:45:59.743 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117477.log 2023-12-04T18:45:59.744 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117626.log: 56.6% -- replaced with /var/log/ceph/cluster1-client.mirror.117626.log.gz 2023-12-04T18:45:59.744 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107135.log 2023-12-04T18:45:59.744 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117477.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117477.log.gz 2023-12-04T18:45:59.744 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104212.log 2023-12-04T18:45:59.746 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107135.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107135.log.gz 2023-12-04T18:45:59.746 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111569.log 2023-12-04T18:45:59.746 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104212.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104212.log.gz 2023-12-04T18:45:59.746 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96852.log 2023-12-04T18:45:59.747 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111569.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111569.log.gz 2023-12-04T18:45:59.748 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112819.log 2023-12-04T18:45:59.749 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96852.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.96852.log.gz 2023-12-04T18:45:59.749 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110157.log 2023-12-04T18:45:59.750 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112819.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112819.log.gz 2023-12-04T18:45:59.751 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93630.log 2023-12-04T18:45:59.751 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110157.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110157.log.gz 2023-12-04T18:45:59.752 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102451.log 2023-12-04T18:45:59.753 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93630.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.93630.log.gz 2023-12-04T18:45:59.753 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98389.log 2023-12-04T18:45:59.755 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102451.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102451.log.gz 2023-12-04T18:45:59.755 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113809.log 2023-12-04T18:45:59.760 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98389.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98389.log.gz 2023-12-04T18:45:59.761 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115205.log 2023-12-04T18:45:59.762 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113809.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113809.log.gz 2023-12-04T18:45:59.763 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117439.log 2023-12-04T18:45:59.764 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115205.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115205.log.gz 2023-12-04T18:45:59.764 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116485.log 2023-12-04T18:45:59.765 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117439.log: 67.9% -- replaced with /var/log/ceph/cluster1-client.mirror.117439.log.gz 2023-12-04T18:45:59.765 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96828.log 2023-12-04T18:45:59.766 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116485.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116485.log.gz 2023-12-04T18:45:59.767 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92875.log 2023-12-04T18:45:59.768 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96828.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.96828.log.gz 2023-12-04T18:45:59.768 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95727.log 2023-12-04T18:45:59.769 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.92875.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.92875.log.gz 2023-12-04T18:45:59.769 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90580.log 2023-12-04T18:45:59.774 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95727.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104254.log.gz 2023-12-04T18:45:59.779 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95727.log.gz 2023-12-04T18:45:59.779 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114132.log 2023-12-04T18:45:59.779 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90580.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90580.log.gz 2023-12-04T18:45:59.779 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99498.log 2023-12-04T18:45:59.779 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114132.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114132.log.gz 2023-12-04T18:45:59.779 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109121.log 2023-12-04T18:45:59.780 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99498.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99498.log.gz 2023-12-04T18:45:59.780 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94917.log 2023-12-04T18:45:59.780 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109121.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109121.log.gz 2023-12-04T18:45:59.780 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100420.log 2023-12-04T18:45:59.781 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94917.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94917.log.gz 2023-12-04T18:45:59.781 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108523.log 2023-12-04T18:45:59.781 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100420.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100420.log.gz 2023-12-04T18:45:59.781 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99217.log 2023-12-04T18:45:59.782 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108523.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108523.log.gz 2023-12-04T18:45:59.783 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94155.log 2023-12-04T18:45:59.784 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99217.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99217.log.gz 2023-12-04T18:45:59.784 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102981.log.gz 2023-12-04T18:45:59.784 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105511.log 2023-12-04T18:45:59.786 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114525.log 2023-12-04T18:45:59.786 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105511.log: /var/log/ceph/cluster1-client.mirror.94155.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.94155.log.gz 2023-12-04T18:45:59.786 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105511.log.gz 2023-12-04T18:45:59.786 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106653.log 2023-12-04T18:45:59.786 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108543.log 2023-12-04T18:45:59.789 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114525.log: /var/log/ceph/cluster1-client.mirror.106653.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114525.log.gz 2023-12-04T18:45:59.789 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100506.log 2023-12-04T18:45:59.789 INFO:teuthology.orchestra.run.smithi176.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106653.log.gz 2023-12-04T18:45:59.789 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108543.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85722.log 2023-12-04T18:45:59.789 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100506.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108543.log.gz 2023-12-04T18:45:59.789 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107198.log 2023-12-04T18:45:59.790 INFO:teuthology.orchestra.run.smithi176.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100506.log.gz 2023-12-04T18:45:59.790 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.85722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116325.log 2023-12-04T18:45:59.790 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107198.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85722.log.gz 2023-12-04T18:45:59.790 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108338.log 2023-12-04T18:45:59.794 INFO:teuthology.orchestra.run.smithi176.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107198.log.gz 2023-12-04T18:45:59.795 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110109.log 2023-12-04T18:45:59.795 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116325.log.gz 2023-12-04T18:45:59.795 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108338.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99518.log 2023-12-04T18:45:59.795 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.110109.log: -- replaced with /var/log/ceph/cluster1-client.mirror.108338.log.gz 2023-12-04T18:45:59.795 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110109.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92016.log 2023-12-04T18:45:59.795 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.795 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99518.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91965.log 2023-12-04T18:45:59.795 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.92016.log: 56.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92016.log.gz 2023-12-04T18:45:59.796 INFO:teuthology.orchestra.run.smithi176.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99518.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118199.log 2023-12-04T18:45:59.796 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.796 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105351.log 2023-12-04T18:45:59.796 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118199.log: /var/log/ceph/cluster2-client.mirror.91965.log: 54.0% -- replaced with /var/log/ceph/cluster2-client.mirror.91965.log.gz 2023-12-04T18:45:59.796 INFO:teuthology.orchestra.run.smithi176.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118199.log.gz 2023-12-04T18:45:59.796 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97058.log 2023-12-04T18:45:59.796 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102629.log 2023-12-04T18:45:59.802 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.105351.log: /var/log/ceph/cluster1-client.mirror.97058.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97058.log.gz 2023-12-04T18:45:59.802 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116705.log 2023-12-04T18:45:59.802 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102629.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102629.log.gz 2023-12-04T18:45:59.802 INFO:teuthology.orchestra.run.smithi176.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105351.log.gz 2023-12-04T18:45:59.802 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117404.log 2023-12-04T18:45:59.802 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.116705.log: -- /var/log/ceph/cluster1-client.mirror.115905.log 2023-12-04T18:45:59.803 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116705.log.gz 2023-12-04T18:45:59.803 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117404.log: 48.2% -- replaced with /var/log/ceph/cluster1-client.mirror.117404.log.gz 2023-12-04T18:45:59.803 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117366.log 2023-12-04T18:45:59.803 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113966.log 2023-12-04T18:45:59.803 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115905.log.gz 2023-12-04T18:45:59.803 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117366.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117366.log.gz 2023-12-04T18:45:59.803 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109967.log 2023-12-04T18:45:59.803 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113966.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.95958.log 2023-12-04T18:45:59.803 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109967.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109967.log.gz 2023-12-04T18:45:59.803 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115265.log 2023-12-04T18:45:59.804 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.1.95958.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113966.log.gz 2023-12-04T18:45:59.804 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96165.log 2023-12-04T18:45:59.804 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115265.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115265.log.gz 2023-12-04T18:45:59.807 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90030.log 2023-12-04T18:45:59.807 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96165.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.96165.log.gz 2023-12-04T18:45:59.808 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108061.log 2023-12-04T18:45:59.808 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90030.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90030.log.gz 2023-12-04T18:45:59.808 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101251.log.gz 2023-12-04T18:45:59.808 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114685.log 2023-12-04T18:45:59.808 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108061.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108061.log.gz 2023-12-04T18:45:59.808 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103413.log 2023-12-04T18:45:59.808 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101447.log 2023-12-04T18:45:59.808 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114685.log: /var/log/ceph/cluster2-client.mirror.103413.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114685.log.gz 2023-12-04T18:45:59.808 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107762.log 2023-12-04T18:45:59.809 INFO:teuthology.orchestra.run.smithi176.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103413.log.gz/var/log/ceph/cluster2-client.mirror.101447.log: 2023-12-04T18:45:59.810 INFO:teuthology.orchestra.run.smithi176.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101447.log.gz 2023-12-04T18:45:59.810 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105443.log 2023-12-04T18:45:59.810 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101684.log 2023-12-04T18:45:59.810 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107762.log: /var/log/ceph/cluster2-client.mirror.105443.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107762.log.gz 2023-12-04T18:45:59.811 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118187.log 2023-12-04T18:45:59.811 INFO:teuthology.orchestra.run.smithi176.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.105443.log.gz 2023-12-04T18:45:59.811 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101684.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101684.log.gz 2023-12-04T18:45:59.812 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118109.log 2023-12-04T18:45:59.812 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106424.log 2023-12-04T18:45:59.812 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118187.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118187.log.gz 2023-12-04T18:45:59.812 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118109.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118109.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96341.log 2023-12-04T18:45:59.812 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.813 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106424.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.106424.log.gz 2023-12-04T18:45:59.813 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102730.log 2023-12-04T18:45:59.813 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105604.log 2023-12-04T18:45:59.826 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.96341.log: /var/log/ceph/cluster2-client.mirror.102730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98948.log 2023-12-04T18:45:59.828 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.105604.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105604.log.gz 2023-12-04T18:45:59.839 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111384.log 2023-12-04T18:45:59.844 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98948.log: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.95958.log.gz 2023-12-04T18:45:59.844 INFO:teuthology.orchestra.run.smithi176.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98948.log.gz 2023-12-04T18:45:59.845 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97436.log 2023-12-04T18:45:59.846 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111384.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111384.log.gz 2023-12-04T18:45:59.846 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104195.log 2023-12-04T18:45:59.852 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.97436.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97436.log.gz 2023-12-04T18:45:59.852 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107156.log 2023-12-04T18:45:59.854 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104195.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104195.log.gz 2023-12-04T18:45:59.854 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114565.log 2023-12-04T18:45:59.856 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107156.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107156.log.gz 2023-12-04T18:45:59.856 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107039.log 2023-12-04T18:45:59.857 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114565.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114565.log.gz 2023-12-04T18:45:59.858 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111477.log 2023-12-04T18:45:59.859 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107039.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107039.log.gz 2023-12-04T18:45:59.859 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104432.log 2023-12-04T18:45:59.861 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111477.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.111477.log.gz 2023-12-04T18:45:59.861 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114585.log 2023-12-04T18:45:59.863 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104432.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104432.log.gz 2023-12-04T18:45:59.863 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96940.log 2023-12-04T18:45:59.864 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114585.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114585.log.gz 2023-12-04T18:45:59.864 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103852.log 2023-12-04T18:45:59.868 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.96940.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104175.log.gz 2023-12-04T18:45:59.869 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96940.log.gz 2023-12-04T18:45:59.871 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106716.log 2023-12-04T18:45:59.871 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.103852.log: -- /var/log/ceph/cluster2-client.mirror.104037.log 2023-12-04T18:45:59.871 INFO:teuthology.orchestra.run.smithi176.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103852.log.gz 2023-12-04T18:45:59.873 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106716.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106716.log.gz 2023-12-04T18:45:59.873 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109326.log 2023-12-04T18:45:59.873 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104037.log: gzip -5 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104037.log.gz 2023-12-04T18:45:59.873 INFO:teuthology.orchestra.run.smithi176.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.113672.log 2023-12-04T18:45:59.873 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103911.log 2023-12-04T18:45:59.873 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109326.log: /var/log/ceph/cluster1-client.mirror.113672.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109326.log.gz 2023-12-04T18:45:59.874 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113672.log.gz 2023-12-04T18:45:59.875 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.105870.log 2023-12-04T18:45:59.875 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101667.log 2023-12-04T18:45:59.887 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.1.105870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110844.log 2023-12-04T18:45:59.887 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101667.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101667.log.gz 2023-12-04T18:45:59.898 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102829.log 2023-12-04T18:45:59.898 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110844.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110844.log.gz 2023-12-04T18:45:59.911 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115505.log 2023-12-04T18:45:59.911 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102829.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102829.log.gz 2023-12-04T18:45:59.915 INFO:teuthology.orchestra.run.smithi176.stderr: 95.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.105870.log.gz 2023-12-04T18:45:59.918 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117165.log 2023-12-04T18:45:59.919 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115505.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115505.log.gz 2023-12-04T18:45:59.920 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105833.log 2023-12-04T18:45:59.921 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117165.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92785.log.gz 2023-12-04T18:45:59.921 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117165.log.gz 2023-12-04T18:45:59.921 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111726.log 2023-12-04T18:45:59.922 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.105833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103038.log 2023-12-04T18:45:59.924 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111726.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105833.log.gz 2023-12-04T18:45:59.924 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115045.log 2023-12-04T18:45:59.924 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103038.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111726.log.gz 2023-12-04T18:45:59.924 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110085.log 2023-12-04T18:45:59.924 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103038.log.gz 2023-12-04T18:45:59.926 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115045.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115045.log.gz 2023-12-04T18:45:59.926 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98260.log 2023-12-04T18:45:59.926 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.110085.log: --verbose -- /var/log/ceph/cluster1-client.mirror.115125.log 2023-12-04T18:45:59.927 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110085.log.gz 2023-12-04T18:45:59.927 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100851.log 2023-12-04T18:45:59.928 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115125.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115125.log.gz 2023-12-04T18:45:59.928 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114246.log 2023-12-04T18:45:59.928 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100851.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100851.log.gz 2023-12-04T18:45:59.931 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108085.log 2023-12-04T18:45:59.931 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.114246.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.114246.log.gz 2023-12-04T18:45:59.931 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109555.log 2023-12-04T18:45:59.931 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108085.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98260.log.gz 2023-12-04T18:45:59.931 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108085.log.gz 2023-12-04T18:45:59.931 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109145.log 2023-12-04T18:45:59.933 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92901.log 2023-12-04T18:45:59.933 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109555.log: /var/log/ceph/cluster1-client.mirror.109145.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109555.log.gz 2023-12-04T18:45:59.933 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108663.log 2023-12-04T18:45:59.933 INFO:teuthology.orchestra.run.smithi176.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109145.log.gz 2023-12-04T18:45:59.933 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.92901.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.92901.log.gz 2023-12-04T18:45:59.934 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105210.log 2023-12-04T18:45:59.935 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116625.log 2023-12-04T18:45:59.935 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108663.log: /var/log/ceph/cluster2-client.mirror.105210.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108663.log.gz 2023-12-04T18:45:59.935 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105210.log.gz 2023-12-04T18:45:59.935 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109374.log 2023-12-04T18:45:59.936 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116625.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.116625.log.gz 2023-12-04T18:45:59.936 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114985.log 2023-12-04T18:45:59.936 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102708.log 2023-12-04T18:45:59.936 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109374.log: /var/log/ceph/cluster1-client.mirror.114985.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109374.log.gz 2023-12-04T18:45:59.937 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114985.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102513.log 2023-12-04T18:45:59.937 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.937 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102708.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102708.log.gz 2023-12-04T18:45:59.938 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101842.log 2023-12-04T18:45:59.939 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117265.log 2023-12-04T18:45:59.939 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102513.log: /var/log/ceph/cluster2-client.mirror.101842.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102513.log.gz 2023-12-04T18:45:59.939 INFO:teuthology.orchestra.run.smithi176.stderr: 86.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101842.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99357.log 2023-12-04T18:45:59.939 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:45:59.939 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.111243.log 2023-12-04T18:45:59.940 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117265.log.gz 2023-12-04T18:45:59.944 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114945.log 2023-12-04T18:45:59.944 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.3.111243.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99357.log.gz 2023-12-04T18:45:59.944 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101130.log 2023-12-04T18:45:59.944 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114945.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114945.log.gz 2023-12-04T18:45:59.944 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102550.log 2023-12-04T18:45:59.944 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101130.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101130.log.gz 2023-12-04T18:45:59.945 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90244.log 2023-12-04T18:45:59.945 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102550.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102550.log.gz 2023-12-04T18:45:59.949 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109511.log 2023-12-04T18:45:59.949 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90244.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90244.log.gz 2023-12-04T18:45:59.949 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105005.log 2023-12-04T18:45:59.949 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109511.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109511.log.gz 2023-12-04T18:45:59.950 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117865.log 2023-12-04T18:45:59.950 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105005.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105005.log.gz 2023-12-04T18:45:59.950 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99277.log 2023-12-04T18:45:59.950 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.117865.log: 30.7% -- replaced with /var/log/ceph/cluster1-client.admin.117865.log.gz 2023-12-04T18:45:59.950 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114605.log 2023-12-04T18:45:59.950 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99277.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99277.log.gz 2023-12-04T18:45:59.952 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104116.log 2023-12-04T18:45:59.953 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114605.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114605.log.gz 2023-12-04T18:45:59.953 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103218.log 2023-12-04T18:45:59.953 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104116.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104116.log.gz 2023-12-04T18:45:59.953 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100745.log 2023-12-04T18:45:59.957 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103218.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101410.log.gz 2023-12-04T18:45:59.957 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100123.log 2023-12-04T18:45:59.957 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100745.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.100745.log.gz 2023-12-04T18:45:59.957 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118372.log 2023-12-04T18:45:59.957 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100123.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100123.log.gz 2023-12-04T18:45:59.957 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104654.log 2023-12-04T18:45:59.957 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118372.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118372.log.gz 2023-12-04T18:45:59.957 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110431.log 2023-12-04T18:45:59.959 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104654.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104654.log.gz 2023-12-04T18:45:59.959 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104676.log 2023-12-04T18:45:59.960 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110431.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110431.log.gz 2023-12-04T18:45:59.960 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114805.log 2023-12-04T18:45:59.971 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116385.log 2023-12-04T18:45:59.972 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114805.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114805.log.gz 2023-12-04T18:45:59.976 INFO:teuthology.orchestra.run.smithi176.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.3.111243.log.gz 2023-12-04T18:45:59.976 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107832.log 2023-12-04T18:45:59.978 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116385.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116385.log.gz 2023-12-04T18:45:59.978 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118024.log 2023-12-04T18:45:59.978 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107832.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.107832.log.gz 2023-12-04T18:45:59.980 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112797.log 2023-12-04T18:45:59.980 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118024.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118024.log.gz 2023-12-04T18:45:59.980 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111025.log 2023-12-04T18:45:59.981 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112797.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112797.log.gz 2023-12-04T18:45:59.981 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100403.log 2023-12-04T18:45:59.983 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111025.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.111025.log.gz 2023-12-04T18:45:59.984 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107627.log 2023-12-04T18:45:59.984 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100403.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100403.log.gz 2023-12-04T18:45:59.984 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97889.log 2023-12-04T18:45:59.985 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107627.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107627.log.gz 2023-12-04T18:45:59.985 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109166.log 2023-12-04T18:45:59.990 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.97889.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97889.log.gz 2023-12-04T18:45:59.991 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116565.log 2023-12-04T18:45:59.992 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109166.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109166.log.gz 2023-12-04T18:45:59.994 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.112390.log 2023-12-04T18:45:59.994 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116565.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116565.log.gz 2023-12-04T18:45:59.994 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107812.log 2023-12-04T18:46:00.004 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.0.112390.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91106.log 2023-12-04T18:46:00.005 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107812.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107812.log.gz 2023-12-04T18:46:00.017 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101308.log 2023-12-04T18:46:00.017 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.91106.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.91106.log.gz 2023-12-04T18:46:00.030 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102530.log 2023-12-04T18:46:00.031 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101308.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101308.log.gz 2023-12-04T18:46:00.035 INFO:teuthology.orchestra.run.smithi176.stderr: 95.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.112390.log.gz 2023-12-04T18:46:00.035 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109925.log 2023-12-04T18:46:00.035 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102530.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102530.log.gz 2023-12-04T18:46:00.036 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100262.log 2023-12-04T18:46:00.039 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109925.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109925.log.gz 2023-12-04T18:46:00.039 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108796.log 2023-12-04T18:46:00.039 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100262.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100262.log.gz 2023-12-04T18:46:00.041 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96254.log 2023-12-04T18:46:00.041 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108796.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108796.log.gz 2023-12-04T18:46:00.041 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.86034.log 2023-12-04T18:46:00.047 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96254.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96254.log.gz 2023-12-04T18:46:00.047 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93343.log 2023-12-04T18:46:00.047 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.86034.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.86034.log.gz 2023-12-04T18:46:00.048 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106694.log 2023-12-04T18:46:00.049 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93343.log: 74.0% -- replaced with /var/log/ceph/cluster2-client.mirror.93343.log.gz 2023-12-04T18:46:00.049 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118341.log 2023-12-04T18:46:00.051 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106694.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102493.log.gz 2023-12-04T18:46:00.052 INFO:teuthology.orchestra.run.smithi176.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.106694.log.gz 2023-12-04T18:46:00.052 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111956.log 2023-12-04T18:46:00.052 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.118341.log: --verbose -- /var/log/ceph/cluster2-client.mirror.105098.log 2023-12-04T18:46:00.052 INFO:teuthology.orchestra.run.smithi176.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118341.log.gz 2023-12-04T18:46:00.052 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118123.log 2023-12-04T18:46:00.055 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111956.log: /var/log/ceph/cluster2-client.mirror.105098.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111956.log.gz 2023-12-04T18:46:00.055 INFO:teuthology.orchestra.run.smithi176.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105098.log.gz 2023-12-04T18:46:00.055 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108431.log 2023-12-04T18:46:00.055 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102572.log.gz/var/log/ceph/cluster2-client.mirror.118123.log: 2023-12-04T18:46:00.055 INFO:teuthology.orchestra.run.smithi176.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118123.log.gz 2023-12-04T18:46:00.055 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105371.log 2023-12-04T18:46:00.056 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108955.log 2023-12-04T18:46:00.056 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108431.log: /var/log/ceph/cluster1-client.mirror.105371.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108431.log.gz 2023-12-04T18:46:00.056 INFO:teuthology.orchestra.run.smithi176.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105371.log.gz 2023-12-04T18:46:00.059 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106766.log 2023-12-04T18:46:00.059 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117789.log 2023-12-04T18:46:00.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106766.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111706.log 2023-12-04T18:46:00.060 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.108955.log.gz 2023-12-04T18:46:00.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117789.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.106766.log.gz 2023-12-04T18:46:00.060 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117789.log.gz 2023-12-04T18:46:00.060 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107740.log 2023-12-04T18:46:00.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111706.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117105.log 2023-12-04T18:46:00.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117588.log 2023-12-04T18:46:00.064 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117105.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.117105.log.gz 2023-12-04T18:46:00.064 INFO:teuthology.orchestra.run.smithi176.stderr: 92.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100462.log 2023-12-04T18:46:00.064 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.111706.log.gz 2023-12-04T18:46:00.064 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117588.log: gzip -5 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107740.log.gz 2023-12-04T18:46:00.064 INFO:teuthology.orchestra.run.smithi176.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.113166.log 2023-12-04T18:46:00.064 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100462.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117588.log.gz 2023-12-04T18:46:00.064 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96617.log 2023-12-04T18:46:00.065 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104133.log 2023-12-04T18:46:00.065 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96617.log: /var/log/ceph/cluster1-client.mirror.113166.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113166.log.gz 2023-12-04T18:46:00.065 INFO:teuthology.orchestra.run.smithi176.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.96617.log.gz 2023-12-04T18:46:00.065 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98495.log 2023-12-04T18:46:00.065 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115385.log 2023-12-04T18:46:00.065 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104133.log: /var/log/ceph/cluster1-client.mirror.98495.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104133.log.gz 2023-12-04T18:46:00.065 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108473.log 2023-12-04T18:46:00.070 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115385.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115385.log.gz 2023-12-04T18:46:00.070 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91657.log 2023-12-04T18:46:00.070 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108473.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108473.log.gz 2023-12-04T18:46:00.071 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101605.log 2023-12-04T18:46:00.071 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.102730.log.gz 2023-12-04T18:46:00.071 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.91657.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91657.log.gz 2023-12-04T18:46:00.071 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107856.log 2023-12-04T18:46:00.071 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93887.log 2023-12-04T18:46:00.071 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107856.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101605.log.gz 2023-12-04T18:46:00.071 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98495.log.gz 2023-12-04T18:46:00.071 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107856.log.gz 2023-12-04T18:46:00.072 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99997.log 2023-12-04T18:46:00.072 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117205.log 2023-12-04T18:46:00.072 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111162.log 2023-12-04T18:46:00.072 INFO:teuthology.orchestra.run.smithi176.stderr: 89.3% 84.1% -- replaced with /var/log/ceph/cluster2-client.mirror.99997.log.gz 2023-12-04T18:46:00.072 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.93887.log.gz 2023-12-04T18:46:00.072 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117205.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106075.log 2023-12-04T18:46:00.072 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111162.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117205.log.gz 2023-12-04T18:46:00.072 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98925.log 2023-12-04T18:46:00.077 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106557.log 2023-12-04T18:46:00.077 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106075.log: /var/log/ceph/cluster1-client.mirror.98925.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106075.log.gz 2023-12-04T18:46:00.077 INFO:teuthology.orchestra.run.smithi176.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111162.log.gz 2023-12-04T18:46:00.078 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111589.log 2023-12-04T18:46:00.078 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99417.log 2023-12-04T18:46:00.078 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106557.log.gz 2023-12-04T18:46:00.078 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93719.log 2023-12-04T18:46:00.078 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111589.log.gz 2023-12-04T18:46:00.078 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99417.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103275.log 2023-12-04T18:46:00.078 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93719.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99417.log.gz 2023-12-04T18:46:00.078 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106304.log 2023-12-04T18:46:00.078 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103275.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103275.log.gz 2023-12-04T18:46:00.079 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100548.log 2023-12-04T18:46:00.079 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106304.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106304.log.gz 2023-12-04T18:46:00.079 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98925.log.gz 2023-12-04T18:46:00.079 INFO:teuthology.orchestra.run.smithi176.stderr: -- /var/log/ceph/cluster1-client.mirror.96297.log 2023-12-04T18:46:00.079 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115545.log 2023-12-04T18:46:00.080 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96297.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93719.log.gz 2023-12-04T18:46:00.080 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106786.log 2023-12-04T18:46:00.083 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115545.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115545.log.gz 2023-12-04T18:46:00.083 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91300.log 2023-12-04T18:46:00.083 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106786.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106786.log.gz 2023-12-04T18:46:00.083 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105393.log 2023-12-04T18:46:00.083 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.91300.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91300.log.gz 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96297.log.gz 2023-12-04T18:46:00.084 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:00.084 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105788.log 2023-12-04T18:46:00.085 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112961.log 2023-12-04T18:46:00.085 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.105393.log: /var/log/ceph/cluster1-client.mirror.105788.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105393.log.gz 2023-12-04T18:46:00.085 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105788.log.gz 2023-12-04T18:46:00.085 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111121.log 2023-12-04T18:46:00.085 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117985.log 2023-12-04T18:46:00.086 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112961.log: /var/log/ceph/cluster1-client.mirror.111121.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112961.log.gz 2023-12-04T18:46:00.089 INFO:teuthology.orchestra.run.smithi176.stderr:gzip 87.8% -5 --verbose -- /var/log/ceph/cluster2-client.admin.85895.log 2023-12-04T18:46:00.090 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.111121.log.gz 2023-12-04T18:46:00.090 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.117985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114825.log 2023-12-04T18:46:00.090 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.85895.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117985.log.gz 2023-12-04T18:46:00.090 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85895.log.gz 2023-12-04T18:46:00.090 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96189.log 2023-12-04T18:46:00.090 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113026.log 2023-12-04T18:46:00.090 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114825.log: /var/log/ceph/cluster1-client.mirror.96189.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114825.log.gz 2023-12-04T18:46:00.090 INFO:teuthology.orchestra.run.smithi176.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.96189.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102276.log 2023-12-04T18:46:00.091 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:00.091 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96682.log 2023-12-04T18:46:00.091 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113026.log: /var/log/ceph/cluster2-client.mirror.102276.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102276.log.gz 2023-12-04T18:46:00.091 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104353.log 2023-12-04T18:46:00.091 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.96682.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96682.log.gz 2023-12-04T18:46:00.094 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103576.log 2023-12-04T18:46:00.094 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104353.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104353.log.gz 2023-12-04T18:46:00.094 INFO:teuthology.orchestra.run.smithi176.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113026.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117678.log 2023-12-04T18:46:00.094 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:00.095 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103576.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103576.log.gz 2023-12-04T18:46:00.095 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105893.log 2023-12-04T18:46:00.095 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117678.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117678.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107969.log 2023-12-04T18:46:00.095 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:00.095 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113946.log 2023-12-04T18:46:00.097 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105893.log: /var/log/ceph/cluster1-client.mirror.107969.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107969.log.gz 2023-12-04T18:46:00.097 INFO:teuthology.orchestra.run.smithi176.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105893.log.gz 2023-12-04T18:46:00.097 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109603.log 2023-12-04T18:46:00.098 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100834.log 2023-12-04T18:46:00.098 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109603.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109603.log.gz 2023-12-04T18:46:00.098 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100166.log 2023-12-04T18:46:00.098 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100834.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100834.log.gz 2023-12-04T18:46:00.098 INFO:teuthology.orchestra.run.smithi176.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.113946.log.gz 2023-12-04T18:46:00.098 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105190.log 2023-12-04T18:46:00.100 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95039.log 2023-12-04T18:46:00.100 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100166.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100166.log.gz 2023-12-04T18:46:00.100 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.105190.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90538.log 2023-12-04T18:46:00.100 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95039.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.105190.log.gz 2023-12-04T18:46:00.100 INFO:teuthology.orchestra.run.smithi176.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95039.log.gz 2023-12-04T18:46:00.100 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114865.log 2023-12-04T18:46:00.101 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97478.log 2023-12-04T18:46:00.101 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90538.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90538.log.gz 2023-12-04T18:46:00.102 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114865.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99703.log 2023-12-04T18:46:00.102 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114865.log.gz/var/log/ceph/cluster2-client.mirror.97478.log: 2023-12-04T18:46:00.102 INFO:teuthology.orchestra.run.smithi176.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.97478.log.gz 2023-12-04T18:46:00.102 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108684.log 2023-12-04T18:46:00.102 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106099.log 2023-12-04T18:46:00.103 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99703.log: /var/log/ceph/cluster2-client.mirror.108684.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99703.log.gz 2023-12-04T18:46:00.103 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118048.log 2023-12-04T18:46:00.103 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106099.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106099.log.gz 2023-12-04T18:46:00.105 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116845.log 2023-12-04T18:46:00.105 INFO:teuthology.orchestra.run.smithi176.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108684.log.gz 2023-12-04T18:46:00.105 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118048.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118048.log.gz 2023-12-04T18:46:00.105 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113190.log 2023-12-04T18:46:00.105 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102293.log 2023-12-04T18:46:00.108 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116845.log: /var/log/ceph/cluster1-client.mirror.113190.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116845.log.gz 2023-12-04T18:46:00.108 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113190.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117993.log 2023-12-04T18:46:00.108 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:00.108 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102293.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102293.log.gz 2023-12-04T18:46:00.108 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.86102.log 2023-12-04T18:46:00.109 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.117993.log: 8.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107063.log 2023-12-04T18:46:00.109 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.117993.log.gz 2023-12-04T18:46:00.109 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111497.log 2023-12-04T18:46:00.109 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.86102.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.86102.log.gz 2023-12-04T18:46:00.111 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107063.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111308.log 2023-12-04T18:46:00.112 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107063.log.gz 2023-12-04T18:46:00.112 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109419.log 2023-12-04T18:46:00.112 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111308.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111308.log.gz 2023-12-04T18:46:00.112 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93806.log 2023-12-04T18:46:00.112 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109419.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111497.log.gz 2023-12-04T18:46:00.112 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104370.log 2023-12-04T18:46:00.114 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93806.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109419.log.gz 2023-12-04T18:46:00.114 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2023-12-04T18:46:00.115 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104370.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104370.log.gz 2023-12-04T18:46:00.115 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105487.log 2023-12-04T18:46:00.127 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96897.log 2023-12-04T18:46:00.128 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105487.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105487.log.gz 2023-12-04T18:46:00.134 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113238.log 2023-12-04T18:46:00.139 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.96897.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96897.log.gz 2023-12-04T18:46:00.152 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108615.log 2023-12-04T18:46:00.153 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113238.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113238.log.gz 2023-12-04T18:46:00.165 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90156.log 2023-12-04T18:46:00.165 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108615.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108615.log.gz 2023-12-04T18:46:00.181 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114365.log 2023-12-04T18:46:00.181 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90156.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90156.log.gz 2023-12-04T18:46:00.182 INFO:teuthology.orchestra.run.smithi146.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2023-12-04T18:46:00.190 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102313.log 2023-12-04T18:46:00.191 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114365.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114365.log.gz 2023-12-04T18:46:00.201 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115445.log 2023-12-04T18:46:00.202 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102313.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102313.log.gz 2023-12-04T18:46:00.215 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105118.log 2023-12-04T18:46:00.216 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115445.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115445.log.gz 2023-12-04T18:46:00.224 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106352.log 2023-12-04T18:46:00.233 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115845.log 2023-12-04T18:46:00.234 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106352.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106352.log.gz 2023-12-04T18:46:00.236 INFO:teuthology.orchestra.run.smithi176.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105118.log.gz 2023-12-04T18:46:00.242 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104740.log 2023-12-04T18:46:00.255 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115845.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115845.log.gz 2023-12-04T18:46:00.255 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97413.log 2023-12-04T18:46:00.255 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.104740.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.104740.log.gz 2023-12-04T18:46:00.255 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103911.log.gz 2023-12-04T18:46:00.255 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96471.log 2023-12-04T18:46:00.260 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.97413.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97413.log.gz 2023-12-04T18:46:00.262 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108178.log 2023-12-04T18:46:00.262 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.96471.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.96471.log.gz 2023-12-04T18:46:00.262 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110543.log 2023-12-04T18:46:00.264 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108178.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108178.log.gz 2023-12-04T18:46:00.264 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112777.log 2023-12-04T18:46:00.266 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110543.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.110543.log.gz 2023-12-04T18:46:00.266 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112207.log 2023-12-04T18:46:00.268 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112777.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.112777.log.gz 2023-12-04T18:46:00.268 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110567.log 2023-12-04T18:46:00.269 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112207.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112207.log.gz 2023-12-04T18:46:00.270 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116345.log 2023-12-04T18:46:00.271 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110567.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110567.log.gz 2023-12-04T18:46:00.271 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102054.log 2023-12-04T18:46:00.271 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116345.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116345.log.gz 2023-12-04T18:46:00.273 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99893.log 2023-12-04T18:46:00.273 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102054.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102054.log.gz 2023-12-04T18:46:00.276 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100526.log 2023-12-04T18:46:00.276 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96341.log.gz 2023-12-04T18:46:00.276 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99893.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99893.log.gz 2023-12-04T18:46:00.276 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94963.log 2023-12-04T18:46:00.276 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109186.log 2023-12-04T18:46:00.276 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100526.log.gz 2023-12-04T18:46:00.276 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101019.log 2023-12-04T18:46:00.276 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109186.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94963.log.gz 2023-12-04T18:46:00.278 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114905.log 2023-12-04T18:46:00.278 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101019.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101019.log.gz 2023-12-04T18:46:00.278 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91640.log 2023-12-04T18:46:00.278 INFO:teuthology.orchestra.run.smithi176.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.109186.log.gz 2023-12-04T18:46:00.278 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110362.log 2023-12-04T18:46:00.278 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114905.log.gz 2023-12-04T18:46:00.285 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.91640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115865.log 2023-12-04T18:46:00.285 INFO:teuthology.orchestra.run.smithi176.stderr: 53.5% -- replaced with /var/log/ceph/cluster1-client.mirror.91640.log.gz 2023-12-04T18:46:00.285 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110362.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100324.log 2023-12-04T18:46:00.285 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.115865.log: -- replaced with /var/log/ceph/cluster1-client.mirror.110362.log.gz 2023-12-04T18:46:00.285 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112028.log 2023-12-04T18:46:00.285 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100324.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115865.log.gz 2023-12-04T18:46:00.286 INFO:teuthology.orchestra.run.smithi176.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100324.log.gz 2023-12-04T18:46:00.286 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110037.log 2023-12-04T18:46:00.286 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.112028.log: --verbose -- /var/log/ceph/cluster2-client.mirror.104054.log 2023-12-04T18:46:00.286 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110037.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112028.log.gz 2023-12-04T18:46:00.286 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104232.log 2023-12-04T18:46:00.286 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104054.log: -- replaced with /var/log/ceph/cluster2-client.mirror.110037.log.gz 2023-12-04T18:46:00.286 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116165.log 2023-12-04T18:46:00.286 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104232.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104054.log.gz 2023-12-04T18:46:00.287 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104232.log.gz 2023-12-04T18:46:00.287 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93221.log 2023-12-04T18:46:00.287 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116745.log 2023-12-04T18:46:00.287 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116165.log.gz 2023-12-04T18:46:00.287 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93221.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103001.log 2023-12-04T18:46:00.287 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116745.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93221.log.gz 2023-12-04T18:46:00.287 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116745.log.gz 2023-12-04T18:46:00.287 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118088.log 2023-12-04T18:46:00.291 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98016.log 2023-12-04T18:46:00.292 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118088.log: /var/log/ceph/cluster2-client.mirror.103001.log: 8.2% 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103001.log.gz 2023-12-04T18:46:00.292 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.118088.log.gz 2023-12-04T18:46:00.292 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99011.log 2023-12-04T18:46:00.292 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105258.log 2023-12-04T18:46:00.292 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99011.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98016.log.gz 2023-12-04T18:46:00.292 INFO:teuthology.orchestra.run.smithi176.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99011.log.gz 2023-12-04T18:46:00.292 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97909.log 2023-12-04T18:46:00.292 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107176.log 2023-12-04T18:46:00.293 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105258.log.gz 2023-12-04T18:46:00.293 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.97909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93742.log 2023-12-04T18:46:00.293 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107176.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107176.log.gz 2023-12-04T18:46:00.293 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115065.log 2023-12-04T18:46:00.295 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93742.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97909.log.gz 2023-12-04T18:46:00.295 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96073.log 2023-12-04T18:46:00.295 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115065.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115065.log.gz 2023-12-04T18:46:00.297 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2023-12-04T18:46:00.297 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.96073.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96073.log.gz 2023-12-04T18:46:00.297 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113395.log 2023-12-04T18:46:00.297 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93742.log.gz 2023-12-04T18:46:00.298 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105306.log 2023-12-04T18:46:00.298 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113395.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113395.log.gz 2023-12-04T18:46:00.299 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107087.log 2023-12-04T18:46:00.299 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105306.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105306.log.gz 2023-12-04T18:46:00.300 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112048.log 2023-12-04T18:46:00.300 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107087.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107087.log.gz 2023-12-04T18:46:00.301 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113214.log 2023-12-04T18:46:00.301 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112048.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112048.log.gz 2023-12-04T18:46:00.302 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100666.log 2023-12-04T18:46:00.302 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113214.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113214.log.gz 2023-12-04T18:46:00.303 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102592.log 2023-12-04T18:46:00.304 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100666.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100666.log.gz 2023-12-04T18:46:00.304 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114177.log 2023-12-04T18:46:00.304 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102592.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102592.log.gz 2023-12-04T18:46:00.305 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101704.log 2023-12-04T18:46:00.305 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.114177.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114177.log.gz 2023-12-04T18:46:00.306 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117387.log 2023-12-04T18:46:00.306 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101704.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101704.log.gz 2023-12-04T18:46:00.308 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91589.log 2023-12-04T18:46:00.308 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117387.log: 68.1% -- replaced with /var/log/ceph/cluster1-client.mirror.117387.log.gz 2023-12-04T18:46:00.308 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109278.log 2023-12-04T18:46:00.310 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.91589.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.91589.log.gz 2023-12-04T18:46:00.310 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114665.log 2023-12-04T18:46:00.310 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109278.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109278.log.gz 2023-12-04T18:46:00.312 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94179.log 2023-12-04T18:46:00.312 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114665.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114665.log.gz 2023-12-04T18:46:00.312 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98819.log 2023-12-04T18:46:00.314 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94179.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94179.log.gz 2023-12-04T18:46:00.314 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90874.log 2023-12-04T18:46:00.317 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98819.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103218.log.gz 2023-12-04T18:46:00.319 INFO:teuthology.orchestra.run.smithi176.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98819.log.gz 2023-12-04T18:46:00.322 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100649.log 2023-12-04T18:46:00.322 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.90874.log: -- /var/log/ceph/cluster2-client.admin.118291.log 2023-12-04T18:46:00.322 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90874.log.gz 2023-12-04T18:46:00.322 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100649.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101056.log 2023-12-04T18:46:00.322 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.118291.log: 9.2% 79.3% -- replaced with /var/log/ceph/cluster2-client.mirror.100649.log.gz 2023-12-04T18:46:00.322 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.118291.log.gz 2023-12-04T18:46:00.322 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102254.log 2023-12-04T18:46:00.323 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113853.log 2023-12-04T18:46:00.323 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101056.log: /var/log/ceph/cluster2-client.mirror.102254.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101056.log.gz 2023-12-04T18:46:00.324 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100440.log 2023-12-04T18:46:00.324 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113853.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113853.log.gz 2023-12-04T18:46:00.324 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91742.log 2023-12-04T18:46:00.325 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100440.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100440.log.gz 2023-12-04T18:46:00.325 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101172.log 2023-12-04T18:46:00.325 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.91742.log: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.91742.log.gz 2023-12-04T18:46:00.326 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101430.log 2023-12-04T18:46:00.347 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101388.log 2023-12-04T18:46:00.347 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101430.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101430.log.gz 2023-12-04T18:46:00.359 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.101948.log 2023-12-04T18:46:00.359 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101388.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101388.log.gz 2023-12-04T18:46:00.363 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100462.log.gz 2023-12-04T18:46:00.364 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95664.log 2023-12-04T18:46:00.367 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.0.101948.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.101948.log.gz 2023-12-04T18:46:00.368 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108362.log 2023-12-04T18:46:00.373 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95664.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95664.log.gz 2023-12-04T18:46:00.375 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109651.log 2023-12-04T18:46:00.375 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108362.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108362.log.gz 2023-12-04T18:46:00.375 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100203.log 2023-12-04T18:46:00.375 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109651.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109651.log.gz 2023-12-04T18:46:00.377 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107675.log 2023-12-04T18:46:00.377 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100203.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100203.log.gz 2023-12-04T18:46:00.378 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108567.log 2023-12-04T18:46:00.378 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107675.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107675.log.gz 2023-12-04T18:46:00.378 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117567.log 2023-12-04T18:46:00.380 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108567.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108567.log.gz 2023-12-04T18:46:00.380 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109398.log 2023-12-04T18:46:00.380 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117567.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117567.log.gz 2023-12-04T18:46:00.380 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90370.log 2023-12-04T18:46:00.380 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109398.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109398.log.gz 2023-12-04T18:46:00.380 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109439.log 2023-12-04T18:46:00.381 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90370.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90370.log.gz 2023-12-04T18:46:00.381 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105870.log 2023-12-04T18:46:00.383 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109439.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109439.log.gz 2023-12-04T18:46:00.383 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108410.log 2023-12-04T18:46:00.384 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.105870.log: 68.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105870.log.gz 2023-12-04T18:46:00.384 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91227.log 2023-12-04T18:46:00.384 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108410.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108410.log.gz 2023-12-04T18:46:00.385 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99952.log 2023-12-04T18:46:00.386 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.91227.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91227.log.gz 2023-12-04T18:46:00.386 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2023-12-04T18:46:00.403 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100145.log 2023-12-04T18:46:00.407 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2.log: 88.2% -- replaced with /var/log/ceph/cluster2.log.gz 2023-12-04T18:46:00.420 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106234.log 2023-12-04T18:46:00.433 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111408.log 2023-12-04T18:46:00.434 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106234.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106234.log.gz 2023-12-04T18:46:00.441 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117345.log 2023-12-04T18:46:00.441 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111408.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111408.log.gz 2023-12-04T18:46:00.449 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116865.log 2023-12-04T18:46:00.450 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117345.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117345.log.gz 2023-12-04T18:46:00.467 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99910.log 2023-12-04T18:46:00.467 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116865.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116865.log.gz 2023-12-04T18:46:00.475 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105764.log 2023-12-04T18:46:00.476 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99910.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99910.log.gz 2023-12-04T18:46:00.485 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106192.log 2023-12-04T18:46:00.485 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105764.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105764.log.gz 2023-12-04T18:46:00.499 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96803.log 2023-12-04T18:46:00.501 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106192.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106192.log.gz 2023-12-04T18:46:00.516 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94891.log 2023-12-04T18:46:00.517 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96803.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.96803.log.gz 2023-12-04T18:46:00.529 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90158.log 2023-12-04T18:46:00.530 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94891.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.94891.log.gz 2023-12-04T18:46:00.544 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108041.log 2023-12-04T18:46:00.544 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90158.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90158.log.gz 2023-12-04T18:46:00.548 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100548.log.gz 2023-12-04T18:46:00.549 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112349.log 2023-12-04T18:46:00.550 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108041.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.108041.log.gz 2023-12-04T18:46:00.563 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101172.log.gz 2023-12-04T18:46:00.575 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108868.log 2023-12-04T18:46:00.576 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112349.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112349.log.gz 2023-12-04T18:46:00.576 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91239.log 2023-12-04T18:46:00.577 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108868.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108868.log.gz 2023-12-04T18:46:00.577 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101783.log 2023-12-04T18:46:00.579 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.91239.log: 73.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91239.log.gz 2023-12-04T18:46:00.579 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94528.log 2023-12-04T18:46:00.580 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101783.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101783.log.gz 2023-12-04T18:46:00.580 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102939.log 2023-12-04T18:46:00.581 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94528.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94528.log.gz 2023-12-04T18:46:00.582 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98776.log 2023-12-04T18:46:00.583 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102939.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102939.log.gz 2023-12-04T18:46:00.583 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104449.log 2023-12-04T18:46:00.588 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98776.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98776.log.gz 2023-12-04T18:46:00.589 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106400.log 2023-12-04T18:46:00.590 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104449.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104449.log.gz 2023-12-04T18:46:00.591 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116985.log 2023-12-04T18:46:00.591 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106400.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.106400.log.gz 2023-12-04T18:46:00.592 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102116.log 2023-12-04T18:46:00.593 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116985.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116985.log.gz 2023-12-04T18:46:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99437.log 2023-12-04T18:46:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102116.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102116.log.gz 2023-12-04T18:46:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97372.log 2023-12-04T18:46:00.596 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99437.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99437.log.gz 2023-12-04T18:46:00.596 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113759.log 2023-12-04T18:46:00.597 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.97372.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97372.log.gz 2023-12-04T18:46:00.597 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96141.log 2023-12-04T18:46:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113759.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113759.log.gz 2023-12-04T18:46:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107340.log 2023-12-04T18:46:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96141.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.96141.log.gz 2023-12-04T18:46:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80991.log 2023-12-04T18:46:00.602 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107340.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107340.log.gz 2023-12-04T18:46:00.602 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103176.log 2023-12-04T18:46:00.602 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.tmp-client.admin.80991.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80991.log.gz 2023-12-04T18:46:00.603 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116425.log 2023-12-04T18:46:00.604 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103176.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103176.log.gz 2023-12-04T18:46:00.604 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94307.log 2023-12-04T18:46:00.605 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116425.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116425.log.gz 2023-12-04T18:46:00.605 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113351.log 2023-12-04T18:46:00.616 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94307.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104676.log.gz 2023-12-04T18:46:00.617 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97952.log 2023-12-04T18:46:00.619 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113351.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113351.log.gz 2023-12-04T18:46:00.619 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96960.log 2023-12-04T18:46:00.625 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.97952.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.97952.log.gz 2023-12-04T18:46:00.625 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98518.log 2023-12-04T18:46:00.631 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96960.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96960.log.gz 2023-12-04T18:46:00.632 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101229.log 2023-12-04T18:46:00.638 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98518.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98518.log.gz 2023-12-04T18:46:00.638 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111978.log 2023-12-04T18:46:00.639 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101229.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101229.log.gz 2023-12-04T18:46:00.639 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106445.log 2023-12-04T18:46:00.640 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111978.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111978.log.gz 2023-12-04T18:46:00.641 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107928.log 2023-12-04T18:46:00.642 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106445.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.93806.log.gz 2023-12-04T18:46:00.643 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99337.log 2023-12-04T18:46:00.643 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107928.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106445.log.gz 2023-12-04T18:46:00.643 INFO:teuthology.orchestra.run.smithi176.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.107928.log.gz 2023-12-04T18:46:00.646 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110133.log 2023-12-04T18:46:00.646 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103255.log 2023-12-04T18:46:00.646 INFO:teuthology.orchestra.run.smithi176.stderr: 87.7%/var/log/ceph/cluster1-client.mirror.110133.log: -- replaced with /var/log/ceph/cluster2-client.mirror.99337.log.gz 2023-12-04T18:46:00.646 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115525.log 2023-12-04T18:46:00.646 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103255.log: -- replaced with /var/log/ceph/cluster1-client.mirror.110133.log.gz 2023-12-04T18:46:00.646 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110820.log 2023-12-04T18:46:00.647 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115525.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103255.log.gz 2023-12-04T18:46:00.647 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115525.log.gz 2023-12-04T18:46:00.647 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111685.log 2023-12-04T18:46:00.649 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110820.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110820.log.gz 2023-12-04T18:46:00.649 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102133.log 2023-12-04T18:46:00.649 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90958.log 2023-12-04T18:46:00.650 INFO:teuthology.orchestra.run.smithi176.stderr: 87.6%/var/log/ceph/cluster2-client.mirror.102133.log: -- replaced with /var/log/ceph/cluster1-client.mirror.111685.log.gz 2023-12-04T18:46:00.650 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96771.log 2023-12-04T18:46:00.650 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90958.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90958.log.gz 2023-12-04T18:46:00.650 INFO:teuthology.orchestra.run.smithi176.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102133.log.gz 2023-12-04T18:46:00.650 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100183.log 2023-12-04T18:46:00.650 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99686.log 2023-12-04T18:46:00.654 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96771.log: /var/log/ceph/cluster2-client.mirror.100183.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.96771.log.gz 2023-12-04T18:46:00.654 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108820.log 2023-12-04T18:46:00.654 INFO:teuthology.orchestra.run.smithi176.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100183.log.gz 2023-12-04T18:46:00.654 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103080.log 2023-12-04T18:46:00.654 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108820.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99686.log.gz 2023-12-04T18:46:00.655 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112524.log 2023-12-04T18:46:00.655 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108820.log.gz 2023-12-04T18:46:00.655 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100361.log 2023-12-04T18:46:00.655 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112524.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103080.log.gz 2023-12-04T18:46:00.655 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110660.log 2023-12-04T18:46:00.655 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100361.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100361.log.gz 2023-12-04T18:46:00.655 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115645.log 2023-12-04T18:46:00.655 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110660.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.112524.log.gz 2023-12-04T18:46:00.659 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117754.log 2023-12-04T18:46:00.659 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115645.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115645.log.gz 2023-12-04T18:46:00.659 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106487.log 2023-12-04T18:46:00.659 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.117754.log: 2.3% 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.110660.log.gz 2023-12-04T18:46:00.659 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.117754.log.gz 2023-12-04T18:46:00.660 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91674.log 2023-12-04T18:46:00.660 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103376.log 2023-12-04T18:46:00.660 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106487.log: /var/log/ceph/cluster1-client.mirror.91674.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91674.log.gz 2023-12-04T18:46:00.660 INFO:teuthology.orchestra.run.smithi176.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106487.log.gz 2023-12-04T18:46:00.660 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111005.log 2023-12-04T18:46:00.660 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108591.log 2023-12-04T18:46:00.660 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111005.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111005.log.gz 2023-12-04T18:46:00.660 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95457.log 2023-12-04T18:46:00.661 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108591.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108591.log.gz 2023-12-04T18:46:00.662 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93861.log 2023-12-04T18:46:00.662 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95457.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95457.log.gz 2023-12-04T18:46:00.662 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106376.log 2023-12-04T18:46:00.663 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93861.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93861.log.gz 2023-12-04T18:46:00.663 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111914.log 2023-12-04T18:46:00.663 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106376.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106376.log.gz 2023-12-04T18:46:00.664 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103297.log 2023-12-04T18:46:00.664 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111914.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111914.log.gz 2023-12-04T18:46:00.665 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113624.log 2023-12-04T18:46:00.678 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103297.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103809.log 2023-12-04T18:46:00.678 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113624.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113624.log.gz 2023-12-04T18:46:00.685 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99297.log 2023-12-04T18:46:00.685 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103809.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103809.log.gz 2023-12-04T18:46:00.699 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103018.log 2023-12-04T18:46:00.708 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99297.log: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103334.log 2023-12-04T18:46:00.708 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.99297.log.gz 2023-12-04T18:46:00.717 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103018.log: 88.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110868.log 2023-12-04T18:46:00.722 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103334.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103018.log.gz 2023-12-04T18:46:00.726 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98173.log 2023-12-04T18:46:00.727 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110868.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110868.log.gz 2023-12-04T18:46:00.730 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.103334.log.gz 2023-12-04T18:46:00.735 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111519.log 2023-12-04T18:46:00.742 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98173.log: 93.0% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102254.log.gz 2023-12-04T18:46:00.743 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102609.log 2023-12-04T18:46:00.744 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111519.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111519.log.gz 2023-12-04T18:46:00.745 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98150.log 2023-12-04T18:46:00.746 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102609.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102609.log.gz 2023-12-04T18:46:00.746 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103396.log 2023-12-04T18:46:00.753 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98150.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98150.log.gz 2023-12-04T18:46:00.753 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117005.log 2023-12-04T18:46:00.753 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.98173.log.gz 2023-12-04T18:46:00.753 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103396.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103396.log.gz 2023-12-04T18:46:00.755 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95185.log 2023-12-04T18:46:00.755 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117005.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.117005.log.gz 2023-12-04T18:46:00.756 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104291.log 2023-12-04T18:46:00.757 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95185.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95185.log.gz 2023-12-04T18:46:00.757 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93538.log 2023-12-04T18:46:00.758 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104291.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104291.log.gz 2023-12-04T18:46:00.758 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117225.log 2023-12-04T18:46:00.760 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93538.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93538.log.gz 2023-12-04T18:46:00.760 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95503.log 2023-12-04T18:46:00.761 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117225.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.117225.log.gz 2023-12-04T18:46:00.761 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101568.log 2023-12-04T18:46:00.762 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95503.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95503.log.gz 2023-12-04T18:46:00.763 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97617.log 2023-12-04T18:46:00.776 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101568.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116365.log 2023-12-04T18:46:00.777 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.97617.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.97617.log.gz 2023-12-04T18:46:00.790 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116545.log 2023-12-04T18:46:00.791 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116365.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116365.log.gz 2023-12-04T18:46:00.792 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115785.log 2023-12-04T18:46:00.803 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116545.log: 87.4% 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100145.log.gz 2023-12-04T18:46:00.803 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.116545.log.gz 2023-12-04T18:46:00.803 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102809.log 2023-12-04T18:46:00.803 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115785.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115785.log.gz 2023-12-04T18:46:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104981.log 2023-12-04T18:46:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102809.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99952.log.gz 2023-12-04T18:46:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110796.log 2023-12-04T18:46:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104981.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104981.log.gz 2023-12-04T18:46:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102846.log 2023-12-04T18:46:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110796.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110796.log.gz 2023-12-04T18:46:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113048.log 2023-12-04T18:46:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102846.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102846.log.gz 2023-12-04T18:46:00.818 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114745.log 2023-12-04T18:46:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113048.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113048.log.gz 2023-12-04T18:46:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97580.log 2023-12-04T18:46:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114745.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114745.log.gz 2023-12-04T18:46:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101647.log 2023-12-04T18:46:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.97580.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97580.log.gz 2023-12-04T18:46:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112096.log 2023-12-04T18:46:00.830 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94263.log 2023-12-04T18:46:00.830 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112096.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112096.log.gz 2023-12-04T18:46:00.844 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106605.log 2023-12-04T18:46:00.849 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94263.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94263.log.gz 2023-12-04T18:46:00.859 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103789.log 2023-12-04T18:46:00.860 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106605.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106605.log.gz 2023-12-04T18:46:00.867 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103376.log.gz 2023-12-04T18:46:00.872 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91785.log 2023-12-04T18:46:00.874 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103789.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103789.log.gz 2023-12-04T18:46:00.874 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.97617.log 2023-12-04T18:46:00.875 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.91785.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91785.log.gz 2023-12-04T18:46:00.875 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113737.log 2023-12-04T18:46:00.878 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.0.97617.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.97617.log.gz 2023-12-04T18:46:00.879 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113717.log 2023-12-04T18:46:00.880 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113737.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113737.log.gz 2023-12-04T18:46:00.881 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112325.log 2023-12-04T18:46:00.883 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113717.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113717.log.gz 2023-12-04T18:46:00.883 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118177.log 2023-12-04T18:46:00.885 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112325.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112325.log.gz 2023-12-04T18:46:00.885 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116005.log 2023-12-04T18:46:00.885 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118177.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118177.log.gz 2023-12-04T18:46:00.887 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.91121.log 2023-12-04T18:46:00.887 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116005.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103297.log.gz 2023-12-04T18:46:00.888 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116005.log.gz 2023-12-04T18:46:00.888 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115945.log 2023-12-04T18:46:00.888 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.2.91121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114965.log 2023-12-04T18:46:00.888 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115945.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115945.log.gz 2023-12-04T18:46:00.888 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91999.log 2023-12-04T18:46:00.889 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114965.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114965.log.gz 2023-12-04T18:46:00.889 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110955.log 2023-12-04T18:46:00.890 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.91999.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.91999.log.gz 2023-12-04T18:46:00.890 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112756.log 2023-12-04T18:46:00.890 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110955.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110955.log.gz 2023-12-04T18:46:00.891 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110061.log 2023-12-04T18:46:00.892 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112756.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112756.log.gz 2023-12-04T18:46:00.892 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103559.log 2023-12-04T18:46:00.893 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110061.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110061.log.gz 2023-12-04T18:46:00.893 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103618.log 2023-12-04T18:46:00.894 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103559.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103559.log.gz 2023-12-04T18:46:00.894 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93582.log 2023-12-04T18:46:00.915 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97796.log 2023-12-04T18:46:00.916 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93582.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.93582.log.gz 2023-12-04T18:46:00.929 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118301.log 2023-12-04T18:46:00.937 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.97796.log: 89.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96319.log 2023-12-04T18:46:00.937 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.97796.log.gz 2023-12-04T18:46:00.937 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7%/var/log/ceph/cluster2-client.mirror.118301.log: -- replaced with /var/log/ceph/cluster2-client.mirror.94307.log.gz 2023-12-04T18:46:00.939 INFO:teuthology.orchestra.run.smithi176.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118301.log.gz 2023-12-04T18:46:00.940 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107583.log 2023-12-04T18:46:00.940 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.96319.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96319.log.gz 2023-12-04T18:46:00.941 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112277.log 2023-12-04T18:46:00.941 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107583.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.107583.log.gz 2023-12-04T18:46:00.942 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.95000.log 2023-12-04T18:46:00.943 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112277.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112277.log.gz 2023-12-04T18:46:00.943 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93284.log 2023-12-04T18:46:00.948 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.0.95000.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.95000.log.gz 2023-12-04T18:46:00.948 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102335.log 2023-12-04T18:46:00.949 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93284.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93284.log.gz 2023-12-04T18:46:00.950 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107019.log 2023-12-04T18:46:00.960 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102335.log: 94.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.91090.log 2023-12-04T18:46:00.960 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.2.91121.log.gz 2023-12-04T18:46:00.960 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106834.log 2023-12-04T18:46:00.960 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.0.91090.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107019.log.gz 2023-12-04T18:46:00.964 INFO:teuthology.orchestra.run.smithi176.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.91090.log.gz 2023-12-04T18:46:00.973 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117285.log 2023-12-04T18:46:00.975 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106834.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106834.log.gz 2023-12-04T18:46:00.975 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109302.log 2023-12-04T18:46:00.975 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117285.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117285.log.gz 2023-12-04T18:46:00.976 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110410.log 2023-12-04T18:46:00.977 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109302.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109302.log.gz 2023-12-04T18:46:00.978 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118209.log 2023-12-04T18:46:00.981 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110410.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102809.log.gz 2023-12-04T18:46:00.981 INFO:teuthology.orchestra.run.smithi176.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.110410.log.gz 2023-12-04T18:46:00.982 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100341.log 2023-12-04T18:46:00.982 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108270.log 2023-12-04T18:46:00.982 INFO:teuthology.orchestra.run.smithi176.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118209.log.gz 2023-12-04T18:46:00.982 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100341.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96527.log 2023-12-04T18:46:00.982 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108270.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100341.log.gz 2023-12-04T18:46:00.982 INFO:teuthology.orchestra.run.smithi176.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108270.log.gz 2023-12-04T18:46:00.982 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117942.log 2023-12-04T18:46:00.982 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.96527.log: -- /var/log/ceph/cluster1-client.mirror.91708.log 2023-12-04T18:46:00.983 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.117942.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96527.log.gz 2023-12-04T18:46:00.983 INFO:teuthology.orchestra.run.smithi176.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.117942.log.gz 2023-12-04T18:46:00.989 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109531.log 2023-12-04T18:46:00.989 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.91708.log: 56.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110473.log 2023-12-04T18:46:00.989 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.91708.log.gz 2023-12-04T18:46:00.989 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114625.log 2023-12-04T18:46:00.989 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110473.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109531.log.gz 2023-12-04T18:46:00.989 INFO:teuthology.orchestra.run.smithi176.stderr: 88.4%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.110473.log.gz -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116225.log 2023-12-04T18:46:00.989 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:00.990 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114082.log 2023-12-04T18:46:00.990 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116225.log: /var/log/ceph/cluster1-client.mirror.114625.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114625.log.gz 2023-12-04T18:46:00.990 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116225.log.gz 2023-12-04T18:46:00.990 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90496.log 2023-12-04T18:46:00.990 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114222.log 2023-12-04T18:46:00.990 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114082.log: /var/log/ceph/cluster2-client.admin.90496.log: 0.0% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114082.log.gz 2023-12-04T18:46:00.990 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.90496.log.gz 2023-12-04T18:46:00.990 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95771.log 2023-12-04T18:46:00.990 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.114222.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.114222.log.gz 2023-12-04T18:46:00.991 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105330.log 2023-12-04T18:46:00.991 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98561.log 2023-12-04T18:46:00.991 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95771.log: /var/log/ceph/cluster1-client.mirror.105330.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.105330.log.gz 2023-12-04T18:46:00.991 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109856.log 2023-12-04T18:46:00.994 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98561.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98561.log.gz 2023-12-04T18:46:00.994 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91121.log 2023-12-04T18:46:00.995 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109856.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109856.log.gz 2023-12-04T18:46:00.996 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90283.log 2023-12-04T18:46:01.006 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.91121.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94285.log 2023-12-04T18:46:01.006 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90283.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90283.log.gz 2023-12-04T18:46:01.019 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107292.log 2023-12-04T18:46:01.020 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94285.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94285.log.gz 2023-12-04T18:46:01.027 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116105.log 2023-12-04T18:46:01.027 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107292.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107292.log.gz 2023-12-04T18:46:01.035 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104412.log 2023-12-04T18:46:01.036 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116105.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116105.log.gz 2023-12-04T18:46:01.043 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101647.log.gz 2023-12-04T18:46:01.044 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101113.log 2023-12-04T18:46:01.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104412.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92106.log 2023-12-04T18:46:01.060 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101113.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101113.log.gz 2023-12-04T18:46:01.071 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108314.log 2023-12-04T18:46:01.072 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.92106.log: 45.9% -- replaced with /var/log/ceph/cluster2-client.admin.92106.log.gz 2023-12-04T18:46:01.085 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101546.log 2023-12-04T18:46:01.086 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108314.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108314.log.gz 2023-12-04T18:46:01.095 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105053.log 2023-12-04T18:46:01.096 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101546.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101546.log.gz 2023-12-04T18:46:01.103 INFO:teuthology.orchestra.run.smithi176.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.91121.log.gz 2023-12-04T18:46:01.104 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113580.log 2023-12-04T18:46:01.105 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105053.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105053.log.gz 2023-12-04T18:46:01.106 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114785.log 2023-12-04T18:46:01.107 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113580.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113580.log.gz 2023-12-04T18:46:01.107 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106969.log 2023-12-04T18:46:01.108 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114785.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114785.log.gz 2023-12-04T18:46:01.109 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98690.log 2023-12-04T18:46:01.110 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106969.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.106969.log.gz 2023-12-04T18:46:01.111 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115725.log 2023-12-04T18:46:01.116 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98690.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98690.log.gz 2023-12-04T18:46:01.117 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97004.log 2023-12-04T18:46:01.117 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103618.log.gz/var/log/ceph/cluster1-client.mirror.115725.log: 2023-12-04T18:46:01.117 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115725.log.gz 2023-12-04T18:46:01.118 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116205.log 2023-12-04T18:46:01.119 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.97004.log: -- /var/log/ceph/cluster1-client.mirror.98409.log 2023-12-04T18:46:01.120 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116205.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116205.log.gz 2023-12-04T18:46:01.120 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117699.log 2023-12-04T18:46:01.127 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98409.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98409.log.gz 2023-12-04T18:46:01.127 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115605.log 2023-12-04T18:46:01.127 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117699.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117699.log.gz 2023-12-04T18:46:01.127 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107949.log 2023-12-04T18:46:01.128 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115605.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115605.log.gz 2023-12-04T18:46:01.128 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93492.log 2023-12-04T18:46:01.131 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107949.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107949.log.gz 2023-12-04T18:46:01.131 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100489.log 2023-12-04T18:46:01.132 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93492.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93492.log.gz 2023-12-04T18:46:01.132 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107465.log 2023-12-04T18:46:01.134 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100489.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100489.log.gz 2023-12-04T18:46:01.134 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93101.log 2023-12-04T18:46:01.134 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107465.log: 69.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107465.log.gz 2023-12-04T18:46:01.136 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98968.log 2023-12-04T18:46:01.136 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93101.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93101.log.gz 2023-12-04T18:46:01.136 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118441.log 2023-12-04T18:46:01.142 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98968.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98968.log.gz 2023-12-04T18:46:01.142 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.93343.log 2023-12-04T18:46:01.143 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.118441.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.admin.118441.log.gz 2023-12-04T18:46:01.143 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104096.log 2023-12-04T18:46:01.148 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.1.93343.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.93343.log.gz 2023-12-04T18:46:01.148 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95209.log 2023-12-04T18:46:01.161 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97772.log 2023-12-04T18:46:01.162 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95209.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95209.log.gz 2023-12-04T18:46:01.175 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105716.log 2023-12-04T18:46:01.176 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.97772.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97772.log.gz 2023-12-04T18:46:01.177 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94576.log 2023-12-04T18:46:01.179 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.105716.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105716.log.gz 2023-12-04T18:46:01.192 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96213.log 2023-12-04T18:46:01.193 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94576.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94576.log.gz 2023-12-04T18:46:01.199 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92979.log 2023-12-04T18:46:01.200 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96213.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.96213.log.gz 2023-12-04T18:46:01.209 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111184.log 2023-12-04T18:46:01.209 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.92979.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.92979.log.gz 2023-12-04T18:46:01.212 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101568.log.gz 2023-12-04T18:46:01.212 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107248.log 2023-12-04T18:46:01.213 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111184.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111184.log.gz 2023-12-04T18:46:01.213 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101150.log 2023-12-04T18:46:01.215 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107248.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.107248.log.gz 2023-12-04T18:46:01.215 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102372.log 2023-12-04T18:46:01.216 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101150.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101150.log.gz 2023-12-04T18:46:01.216 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118249.log 2023-12-04T18:46:01.217 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102372.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102372.log.gz 2023-12-04T18:46:01.219 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116725.log 2023-12-04T18:46:01.219 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118249.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118249.log.gz 2023-12-04T18:46:01.219 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98862.log 2023-12-04T18:46:01.220 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116725.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116725.log.gz 2023-12-04T18:46:01.220 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115925.log 2023-12-04T18:46:01.226 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98862.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98862.log.gz 2023-12-04T18:46:01.227 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116885.log 2023-12-04T18:46:01.233 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115925.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115925.log.gz 2023-12-04T18:46:01.234 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110772.log 2023-12-04T18:46:01.234 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116885.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116885.log.gz 2023-12-04T18:46:01.234 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103433.log 2023-12-04T18:46:01.234 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110772.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110772.log.gz 2023-12-04T18:46:01.234 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99647.log 2023-12-04T18:46:01.235 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103433.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103433.log.gz 2023-12-04T18:46:01.235 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114885.log 2023-12-04T18:46:01.240 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99647.log: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2023-12-04T18:46:01.241 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112937.log 2023-12-04T18:46:01.243 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114885.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114885.log.gz 2023-12-04T18:46:01.243 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98991.log 2023-12-04T18:46:01.244 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112937.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112937.log.gz 2023-12-04T18:46:01.244 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98733.log 2023-12-04T18:46:01.244 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98991.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98991.log.gz 2023-12-04T18:46:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91565.log 2023-12-04T18:46:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98733.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98733.log.gz 2023-12-04T18:46:01.252 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93762.log 2023-12-04T18:46:01.252 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.91565.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91565.log.gz 2023-12-04T18:46:01.252 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114307.log 2023-12-04T18:46:01.258 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93762.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93762.log.gz 2023-12-04T18:46:01.259 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116645.log 2023-12-04T18:46:01.272 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.114307.log: 91.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112503.log 2023-12-04T18:46:01.273 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116645.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116645.log.gz 2023-12-04T18:46:01.273 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.114307.log.gz 2023-12-04T18:46:01.273 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111360.log 2023-12-04T18:46:01.275 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112503.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112503.log.gz 2023-12-04T18:46:01.275 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112684.log 2023-12-04T18:46:01.276 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111360.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111360.log.gz 2023-12-04T18:46:01.276 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94682.log 2023-12-04T18:46:01.277 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112684.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112684.log.gz 2023-12-04T18:46:01.278 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93035.log 2023-12-04T18:46:01.278 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94682.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94682.log.gz 2023-12-04T18:46:01.279 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113006.log 2023-12-04T18:46:01.280 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93035.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93035.log.gz 2023-12-04T18:46:01.281 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108220.log 2023-12-04T18:46:01.283 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113006.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113006.log.gz 2023-12-04T18:46:01.283 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101288.log 2023-12-04T18:46:01.285 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108220.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108220.log.gz 2023-12-04T18:46:01.285 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111243.log 2023-12-04T18:46:01.287 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101288.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101288.log.gz 2023-12-04T18:46:01.287 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98538.log 2023-12-04T18:46:01.287 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111243.log: 68.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111243.log.gz 2023-12-04T18:46:01.288 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102232.log 2023-12-04T18:46:01.293 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98538.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98538.log.gz 2023-12-04T18:46:01.294 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108704.log 2023-12-04T18:46:01.295 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102232.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102232.log.gz 2023-12-04T18:46:01.295 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112390.log 2023-12-04T18:46:01.297 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108704.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108704.log.gz 2023-12-04T18:46:01.298 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113118.log 2023-12-04T18:46:01.298 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112390.log: 69.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112390.log.gz 2023-12-04T18:46:01.300 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100304.log 2023-12-04T18:46:01.300 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113118.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.113118.log.gz 2023-12-04T18:46:01.300 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91725.log 2023-12-04T18:46:01.313 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108157.log 2023-12-04T18:46:01.314 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.91725.log: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.91725.log.gz 2023-12-04T18:46:01.331 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105696.log 2023-12-04T18:46:01.345 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108157.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108157.log.gz 2023-12-04T18:46:01.358 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108913.log 2023-12-04T18:46:01.359 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.105696.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.105696.log.gz 2023-12-04T18:46:01.365 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116245.log 2023-12-04T18:46:01.367 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108913.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108913.log.gz 2023-12-04T18:46:01.375 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104096.log.gz 2023-12-04T18:46:01.375 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93009.log 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102335.log.gz 2023-12-04T18:46:01.375 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:01.375 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116245.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117900.log 2023-12-04T18:46:01.375 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.116245.log.gz 2023-12-04T18:46:01.375 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95684.log 2023-12-04T18:46:01.376 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117900.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117900.log.gz 2023-12-04T18:46:01.376 INFO:teuthology.orchestra.run.smithi176.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93009.log.gz 2023-12-04T18:46:01.376 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110680.log 2023-12-04T18:46:01.376 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103517.log 2023-12-04T18:46:01.377 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110680.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110680.log.gz 2023-12-04T18:46:01.377 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117125.log 2023-12-04T18:46:01.377 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103517.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103517.log.gz 2023-12-04T18:46:01.377 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103772.log 2023-12-04T18:46:01.378 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117125.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.117125.log.gz 2023-12-04T18:46:01.378 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99051.log 2023-12-04T18:46:01.379 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103772.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103772.log.gz 2023-12-04T18:46:01.379 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98710.log 2023-12-04T18:46:01.380 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95684.log.gz 2023-12-04T18:46:01.380 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110591.log 2023-12-04T18:46:01.381 INFO:teuthology.orchestra.run.smithi176.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99051.log.gz 2023-12-04T18:46:01.381 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98710.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110220.log 2023-12-04T18:46:01.381 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110591.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110591.log.gz 2023-12-04T18:46:01.382 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113925.log 2023-12-04T18:46:01.382 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110220.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110220.log.gz 2023-12-04T18:46:01.382 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98303.log 2023-12-04T18:46:01.383 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113925.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113925.log.gz 2023-12-04T18:46:01.383 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101625.log 2023-12-04T18:46:01.387 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98303.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98710.log.gz 2023-12-04T18:46:01.387 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113259.log 2023-12-04T18:46:01.387 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101625.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101625.log.gz 2023-12-04T18:46:01.388 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117325.log 2023-12-04T18:46:01.390 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113259.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.113259.log.gz 2023-12-04T18:46:01.390 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108844.log 2023-12-04T18:46:01.390 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117325.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98303.log.gz 2023-12-04T18:46:01.393 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100977.log 2023-12-04T18:46:01.393 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117325.log.gz 2023-12-04T18:46:01.393 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118068.log 2023-12-04T18:46:01.393 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108844.log.gz/var/log/ceph/cluster2-client.mirror.100977.log: 2023-12-04T18:46:01.393 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102866.log 2023-12-04T18:46:01.393 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100977.log.gz 2023-12-04T18:46:01.393 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97728.log 2023-12-04T18:46:01.393 INFO:teuthology.orchestra.run.smithi176.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118068.log.gz 2023-12-04T18:46:01.394 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105559.log 2023-12-04T18:46:01.431 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.97728.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102866.log.gz 2023-12-04T18:46:01.431 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104074.log 2023-12-04T18:46:01.431 INFO:teuthology.orchestra.run.smithi176.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97728.log.gz 2023-12-04T18:46:01.431 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97748.log 2023-12-04T18:46:01.431 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104074.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105559.log.gz 2023-12-04T18:46:01.431 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93125.log 2023-12-04T18:46:01.431 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104074.log.gz 2023-12-04T18:46:01.431 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.97748.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95921.log 2023-12-04T18:46:01.432 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6%/var/log/ceph/cluster1-client.mirror.93125.log: -- replaced with /var/log/ceph/cluster2-client.mirror.104412.log.gz 2023-12-04T18:46:01.432 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97748.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.93125.log.gz 2023-12-04T18:46:01.432 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:01.432 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99317.log 2023-12-04T18:46:01.432 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115885.log 2023-12-04T18:46:01.432 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111637.log 2023-12-04T18:46:01.432 INFO:teuthology.orchestra.run.smithi176.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95921.log.gz 2023-12-04T18:46:01.432 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115885.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99317.log.gz 2023-12-04T18:46:01.432 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90622.log 2023-12-04T18:46:01.432 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113142.log 2023-12-04T18:46:01.433 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115885.log.gz 2023-12-04T18:46:01.433 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111637.log.gz 2023-12-04T18:46:01.433 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.90622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114705.log 2023-12-04T18:46:01.433 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90622.log.gz 2023-12-04T18:46:01.433 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110523.log 2023-12-04T18:46:01.433 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.114705.log: -- replaced with /var/log/ceph/cluster1-client.mirror.113142.log.gz 2023-12-04T18:46:01.433 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113600.log 2023-12-04T18:46:01.433 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114705.log.gz 2023-12-04T18:46:01.433 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110270.log 2023-12-04T18:46:01.433 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110523.log: /var/log/ceph/cluster2-client.mirror.113600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117921.log 2023-12-04T18:46:01.434 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110270.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113600.log.gz 2023-12-04T18:46:01.434 INFO:teuthology.orchestra.run.smithi176.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.110523.log.gz 2023-12-04T18:46:01.434 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.110270.log.gz 2023-12-04T18:46:01.434 INFO:teuthology.orchestra.run.smithi176.stderr: /var/log/ceph/cluster1-client.mirror.98796.log 2023-12-04T18:46:01.434 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117921.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117921.log.gz 2023-12-04T18:46:01.434 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2023-12-04T18:46:01.434 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117848.log 2023-12-04T18:46:01.434 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114405.log 2023-12-04T18:46:01.435 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117848.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.117848.log.gz 2023-12-04T18:46:01.435 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113443.log 2023-12-04T18:46:01.435 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114405.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114405.log.gz 2023-12-04T18:46:01.435 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117810.log 2023-12-04T18:46:01.435 INFO:teuthology.orchestra.run.smithi176.stderr: 92.8% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2023-12-04T18:46:01.435 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117720.log 2023-12-04T18:46:01.435 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117810.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113443.log.gz 2023-12-04T18:46:01.435 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117810.log.gz 2023-12-04T18:46:01.435 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89908.log 2023-12-04T18:46:01.435 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99625.log 2023-12-04T18:46:01.436 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117720.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.117720.log.gz 2023-12-04T18:46:01.436 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.89908.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89908.log.gz 2023-12-04T18:46:01.436 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96093.log 2023-12-04T18:46:01.436 INFO:teuthology.orchestra.run.smithi176.stderr:gzip 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98796.log.gz 2023-12-04T18:46:01.436 INFO:teuthology.orchestra.run.smithi176.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96641.log 2023-12-04T18:46:01.436 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.96093.log: /var/log/ceph/cluster2-client.mirror.99625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111842.log 2023-12-04T18:46:01.436 INFO:teuthology.orchestra.run.smithi176.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96093.log.gz 2023-12-04T18:46:01.436 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.96641.log: -- replaced with /var/log/ceph/cluster2-client.mirror.99625.log.gz 2023-12-04T18:46:01.436 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111049.log 2023-12-04T18:46:01.437 INFO:teuthology.orchestra.run.smithi176.stderr: 88.6%/var/log/ceph/cluster1-client.mirror.111842.log: -- replaced with /var/log/ceph/cluster1-client.mirror.96641.log.gz 2023-12-04T18:46:01.437 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114156.log 2023-12-04T18:46:01.437 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111842.log.gz 2023-12-04T18:46:01.437 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94131.log 2023-12-04T18:46:01.437 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111049.log.gz/var/log/ceph/cluster1-client.mirror.114156.log: 2023-12-04T18:46:01.437 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107385.log 2023-12-04T18:46:01.437 INFO:teuthology.orchestra.run.smithi176.stderr: 87.7%/var/log/ceph/cluster1-client.mirror.94131.log: -- replaced with /var/log/ceph/cluster1-client.mirror.114156.log.gz 2023-12-04T18:46:01.437 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115145.log 2023-12-04T18:46:01.437 INFO:teuthology.orchestra.run.smithi176.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.94131.log.gz 2023-12-04T18:46:01.438 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98839.log 2023-12-04T18:46:01.438 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106123.log 2023-12-04T18:46:01.438 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115145.log: /var/log/ceph/cluster1-client.mirror.98839.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115145.log.gz 2023-12-04T18:46:01.438 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114765.log 2023-12-04T18:46:01.438 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106123.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107385.log.gz 2023-12-04T18:46:01.438 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106123.log.gz 2023-12-04T18:46:01.438 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91691.log 2023-12-04T18:46:01.438 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114765.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106284.log 2023-12-04T18:46:01.439 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114765.log.gz 2023-12-04T18:46:01.439 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.91691.log: 57.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98346.log 2023-12-04T18:46:01.439 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.91691.log.gz/var/log/ceph/cluster2-client.mirror.106284.log: 2023-12-04T18:46:01.439 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97004.log.gz 2023-12-04T18:46:01.439 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112072.log 2023-12-04T18:46:01.439 INFO:teuthology.orchestra.run.smithi176.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106284.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106328.log 2023-12-04T18:46:01.439 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:01.439 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98346.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89726.log 2023-12-04T18:46:01.439 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112072.log: /var/log/ceph/cluster1-client.mirror.106328.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112072.log.gz 2023-12-04T18:46:01.440 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106328.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106906.log 2023-12-04T18:46:01.440 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:01.440 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.89726.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89726.log.gz 2023-12-04T18:46:01.440 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98839.log.gz 2023-12-04T18:46:01.440 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115465.log 2023-12-04T18:46:01.440 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106906.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107427.log 2023-12-04T18:46:01.440 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115465.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106906.log.gz 2023-12-04T18:46:01.440 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112120.log 2023-12-04T18:46:01.440 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115465.log.gz 2023-12-04T18:46:01.441 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118165.log 2023-12-04T18:46:01.441 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107427.log: /var/log/ceph/cluster1-client.mirror.112120.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95958.log 2023-12-04T18:46:01.441 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118165.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112120.log.gz 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118165.log.gz 2023-12-04T18:46:01.441 INFO:teuthology.orchestra.run.smithi176.stderr: 88.3% 2023-12-04T18:46:01.441 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.107427.log.gz 2023-12-04T18:46:01.441 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98108.log 2023-12-04T18:46:01.441 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111432.log 2023-12-04T18:46:01.441 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107364.log 2023-12-04T18:46:01.441 INFO:teuthology.orchestra.run.smithi176.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98108.log.gz 2023-12-04T18:46:01.441 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111432.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98346.log.gz 2023-12-04T18:46:01.442 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117515.log 2023-12-04T18:46:01.442 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111432.log.gz 2023-12-04T18:46:01.442 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95113.log 2023-12-04T18:46:01.442 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117515.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100999.log 2023-12-04T18:46:01.442 INFO:teuthology.orchestra.run.smithi176.stderr: 87.7% 49.8% -- replaced with /var/log/ceph/cluster1-client.mirror.117515.log.gz 2023-12-04T18:46:01.442 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.107364.log.gz 2023-12-04T18:46:01.442 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95113.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102471.log 2023-12-04T18:46:01.442 INFO:teuthology.orchestra.run.smithi176.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94504.log 2023-12-04T18:46:01.443 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.95113.log.gz 2023-12-04T18:46:01.443 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100999.log: /var/log/ceph/cluster2-client.mirror.102471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109579.log 2023-12-04T18:46:01.443 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94504.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102471.log.gz 2023-12-04T18:46:01.443 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116905.log 2023-12-04T18:46:01.443 INFO:teuthology.orchestra.run.smithi176.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94504.log.gz 2023-12-04T18:46:01.443 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110338.log 2023-12-04T18:46:01.443 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109579.log.gz 2023-12-04T18:46:01.443 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116905.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103060.log 2023-12-04T18:46:01.443 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116905.log.gz 2023-12-04T18:46:01.444 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115345.log 2023-12-04T18:46:01.444 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110338.log.gz 2023-12-04T18:46:01.444 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118100.log 2023-12-04T18:46:01.444 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103060.log: /var/log/ceph/cluster1-client.mirror.115345.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115345.log.gz 2023-12-04T18:46:01.444 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98581.log 2023-12-04T18:46:01.444 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118100.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118100.log.gz 2023-12-04T18:46:01.444 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95337.log 2023-12-04T18:46:01.446 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98581.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98581.log.gz 2023-12-04T18:46:01.446 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99071.log 2023-12-04T18:46:01.446 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102195.log 2023-12-04T18:46:01.446 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99071.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.99071.log.gz 2023-12-04T18:46:01.458 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112660.log 2023-12-04T18:46:01.459 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102195.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102195.log.gz 2023-12-04T18:46:01.471 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102037.log 2023-12-04T18:46:01.472 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112660.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112660.log.gz 2023-12-04T18:46:01.476 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.95771.log.gz 2023-12-04T18:46:01.478 INFO:teuthology.orchestra.run.smithi176.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.95958.log.gz 2023-12-04T18:46:01.479 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111748.log 2023-12-04T18:46:01.479 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118132.log 2023-12-04T18:46:01.479 INFO:teuthology.orchestra.run.smithi176.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102037.log.gz 2023-12-04T18:46:01.480 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.111748.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111748.log.gz 2023-12-04T18:46:01.480 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110752.log 2023-12-04T18:46:01.481 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104719.log 2023-12-04T18:46:01.481 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118132.log: /var/log/ceph/cluster2-client.mirror.110752.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118132.log.gz 2023-12-04T18:46:01.481 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90284.log 2023-12-04T18:46:01.483 INFO:teuthology.orchestra.run.smithi176.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.110752.log.gz/var/log/ceph/cluster2-client.mirror.104719.log: 2023-12-04T18:46:01.483 INFO:teuthology.orchestra.run.smithi176.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.104719.log.gz 2023-12-04T18:46:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95523.log 2023-12-04T18:46:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103196.log 2023-12-04T18:46:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90284.log: /var/log/ceph/cluster2-client.mirror.95523.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90284.log.gz 2023-12-04T18:46:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117065.log 2023-12-04T18:46:01.484 INFO:teuthology.orchestra.run.smithi176.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95523.log.gz 2023-12-04T18:46:01.485 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103196.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103196.log.gz 2023-12-04T18:46:01.485 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113829.log 2023-12-04T18:46:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108198.log 2023-12-04T18:46:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117065.log: /var/log/ceph/cluster2-client.mirror.113829.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117065.log.gz 2023-12-04T18:46:01.486 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113829.log.gz 2023-12-04T18:46:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116665.log 2023-12-04T18:46:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116085.log 2023-12-04T18:46:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108198.log: /var/log/ceph/cluster1-client.mirror.116665.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116665.log.gz 2023-12-04T18:46:01.487 INFO:teuthology.orchestra.run.smithi176.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108198.log.gz 2023-12-04T18:46:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98475.log 2023-12-04T18:46:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91862.log 2023-12-04T18:46:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116085.log: /var/log/ceph/cluster2-client.mirror.98475.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116085.log.gz 2023-12-04T18:46:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90157.log 2023-12-04T18:46:01.493 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.91862.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98475.log.gz 2023-12-04T18:46:01.496 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99647.log.gz 2023-12-04T18:46:01.508 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115985.log 2023-12-04T18:46:01.508 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90157.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90157.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99477.log 2023-12-04T18:46:01.508 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:01.508 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114270.log 2023-12-04T18:46:01.509 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99477.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115985.log.gz 2023-12-04T18:46:01.509 INFO:teuthology.orchestra.run.smithi176.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99477.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100888.log 2023-12-04T18:46:01.510 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:01.517 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107268.log 2023-12-04T18:46:01.518 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.100888.log: 91.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100888.log.gz 2023-12-04T18:46:01.519 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113488.log 2023-12-04T18:46:01.520 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.107268.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107268.log.gz 2023-12-04T18:46:01.520 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93262.log 2023-12-04T18:46:01.521 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113488.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114270.log.gz 2023-12-04T18:46:01.521 INFO:teuthology.orchestra.run.smithi176.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.113488.log.gz 2023-12-04T18:46:01.523 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110178.log 2023-12-04T18:46:01.523 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.93262.log: --verbose -- /var/log/ceph/cluster2-client.mirror.110290.log 2023-12-04T18:46:01.525 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110178.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110178.log.gz 2023-12-04T18:46:01.525 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103354.log 2023-12-04T18:46:01.526 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110290.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110290.log.gz 2023-12-04T18:46:01.527 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105624.log 2023-12-04T18:46:01.532 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103354.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103354.log.gz 2023-12-04T18:46:01.532 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101948.log 2023-12-04T18:46:01.532 INFO:teuthology.orchestra.run.smithi176.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93262.log.gz 2023-12-04T18:46:01.532 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116965.log 2023-12-04T18:46:01.532 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.101948.log: 84.8% -- replaced with /var/log/ceph/cluster1-client.mirror.101948.log.gz 2023-12-04T18:46:01.532 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109880.log 2023-12-04T18:46:01.533 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116965.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116965.log.gz 2023-12-04T18:46:01.533 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113419.log 2023-12-04T18:46:01.533 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109880.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105624.log.gz 2023-12-04T18:46:01.533 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.109880.log.gz 2023-12-04T18:46:01.533 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98280.log 2023-12-04T18:46:01.533 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103831.log 2023-12-04T18:46:01.533 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113419.log.gz 2023-12-04T18:46:01.534 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98280.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105583.log 2023-12-04T18:46:01.538 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103831.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98280.log.gz 2023-12-04T18:46:01.538 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90706.log 2023-12-04T18:46:01.538 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105583.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.105583.log.gz 2023-12-04T18:46:01.539 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94552.log 2023-12-04T18:46:01.540 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.90706.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90706.log.gz 2023-12-04T18:46:01.540 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111613.log 2023-12-04T18:46:01.540 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94552.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.94552.log.gz 2023-12-04T18:46:01.541 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115965.log 2023-12-04T18:46:01.541 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111613.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111613.log.gz 2023-12-04T18:46:01.542 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103596.log 2023-12-04T18:46:01.543 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115965.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115965.log.gz 2023-12-04T18:46:01.543 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105463.log 2023-12-04T18:46:01.543 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103596.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103596.log.gz 2023-12-04T18:46:01.544 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116185.log 2023-12-04T18:46:01.544 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.105463.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105463.log.gz 2023-12-04T18:46:01.545 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113877.log 2023-12-04T18:46:01.546 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116185.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116185.log.gz 2023-12-04T18:46:01.546 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115705.log 2023-12-04T18:46:01.547 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113877.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113877.log.gz 2023-12-04T18:46:01.547 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109049.log 2023-12-04T18:46:01.547 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115705.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115705.log.gz 2023-12-04T18:46:01.548 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100245.log 2023-12-04T18:46:01.548 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109049.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109049.log.gz 2023-12-04T18:46:01.549 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104888.log 2023-12-04T18:46:01.550 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100245.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100245.log.gz 2023-12-04T18:46:01.550 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109904.log 2023-12-04T18:46:01.551 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.104888.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.104888.log.gz 2023-12-04T18:46:01.551 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99377.log 2023-12-04T18:46:01.551 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109904.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109904.log.gz 2023-12-04T18:46:01.552 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98432.log 2023-12-04T18:46:01.552 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99377.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99377.log.gz 2023-12-04T18:46:01.553 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92927.log 2023-12-04T18:46:01.558 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98432.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98432.log.gz 2023-12-04T18:46:01.559 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104961.log 2023-12-04T18:46:01.559 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.92927.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.92927.log.gz 2023-12-04T18:46:01.560 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118381.log 2023-12-04T18:46:01.560 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104961.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.104961.log.gz 2023-12-04T18:46:01.561 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91283.log 2023-12-04T18:46:01.561 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118381.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118381.log.gz 2023-12-04T18:46:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116925.log 2023-12-04T18:46:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.91283.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.91283.log.gz 2023-12-04T18:46:01.563 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96501.log 2023-12-04T18:46:01.563 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116925.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116925.log.gz 2023-12-04T18:46:01.564 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108133.log 2023-12-04T18:46:01.564 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96501.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.96501.log.gz 2023-12-04T18:46:01.565 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93558.log 2023-12-04T18:46:01.565 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108133.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108133.log.gz 2023-12-04T18:46:01.566 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118228.log 2023-12-04T18:46:01.566 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93558.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.93558.log.gz 2023-12-04T18:46:01.567 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116785.log 2023-12-04T18:46:01.567 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118228.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118228.log.gz 2023-12-04T18:46:01.568 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113301.log 2023-12-04T18:46:01.568 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116785.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116785.log.gz 2023-12-04T18:46:01.569 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2023-12-04T18:46:01.569 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113301.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113301.log.gz 2023-12-04T18:46:01.570 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95547.log 2023-12-04T18:46:01.583 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114725.log 2023-12-04T18:46:01.584 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95547.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95547.log.gz 2023-12-04T18:46:01.589 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.95337.log.gz 2023-12-04T18:46:01.590 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92953.log 2023-12-04T18:46:01.591 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114725.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114725.log.gz 2023-12-04T18:46:01.591 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118546.log 2023-12-04T18:46:01.594 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.92953.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.92953.log.gz 2023-12-04T18:46:01.595 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115085.log 2023-12-04T18:46:01.595 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.118546.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118546.log.gz 2023-12-04T18:46:01.595 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109097.log 2023-12-04T18:46:01.595 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115085.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115085.log.gz 2023-12-04T18:46:01.595 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104698.log 2023-12-04T18:46:01.599 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109097.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109097.log.gz 2023-12-04T18:46:01.599 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117609.log 2023-12-04T18:46:01.599 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104698.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.104698.log.gz 2023-12-04T18:46:01.599 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100103.log 2023-12-04T18:46:01.599 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117609.log: 11.2% -- replaced with /var/log/ceph/cluster1-client.mirror.117609.log.gz 2023-12-04T18:46:01.600 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115105.log 2023-12-04T18:46:01.600 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100103.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100103.log.gz 2023-12-04T18:46:01.600 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104798.log 2023-12-04T18:46:01.600 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115105.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115105.log.gz 2023-12-04T18:46:01.603 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115425.log 2023-12-04T18:46:01.603 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.104798.log: 69.2% -- replaced with /var/log/ceph/cluster1-client.mirror.104798.log.gz 2023-12-04T18:46:01.604 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102787.log 2023-12-04T18:46:01.604 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115425.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115425.log.gz 2023-12-04T18:46:01.604 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98882.log 2023-12-04T18:46:01.604 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102787.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102787.log.gz 2023-12-04T18:46:01.604 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94704.log 2023-12-04T18:46:01.608 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98882.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98882.log.gz 2023-12-04T18:46:01.609 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112616.log 2023-12-04T18:46:01.613 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94704.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100304.log.gz 2023-12-04T18:46:01.617 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97300.log 2023-12-04T18:46:01.618 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112616.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112616.log.gz 2023-12-04T18:46:01.619 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90664.log 2023-12-04T18:46:01.621 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.97300.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97300.log.gz 2023-12-04T18:46:01.621 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94617.log 2023-12-04T18:46:01.621 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.90664.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90664.log.gz 2023-12-04T18:46:01.622 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102153.log 2023-12-04T18:46:01.628 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94617.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.94617.log.gz 2023-12-04T18:46:01.628 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.91227.log 2023-12-04T18:46:01.629 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102153.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102153.log.gz 2023-12-04T18:46:01.629 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109784.log 2023-12-04T18:46:01.635 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.0.91227.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.91227.log.gz 2023-12-04T18:46:01.636 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100686.log 2023-12-04T18:46:01.637 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109784.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109784.log.gz 2023-12-04T18:46:01.637 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116065.log 2023-12-04T18:46:01.638 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100686.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100686.log.gz 2023-12-04T18:46:01.639 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118038.log 2023-12-04T18:46:01.641 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116065.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116065.log.gz 2023-12-04T18:46:01.641 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105029.log 2023-12-04T18:46:01.641 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118038.log: 6.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118038.log.gz 2023-12-04T18:46:01.641 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.91145.log 2023-12-04T18:46:01.643 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105029.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105029.log.gz 2023-12-04T18:46:01.643 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109005.log 2023-12-04T18:46:01.643 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.3.91145.log: 73.1% -- replaced with /var/log/ceph/cluster1-client.mirror.3.91145.log.gz 2023-12-04T18:46:01.643 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102922.log 2023-12-04T18:46:01.644 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109005.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109005.log.gz 2023-12-04T18:46:01.647 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89824.log 2023-12-04T18:46:01.647 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102922.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102922.log.gz 2023-12-04T18:46:01.647 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115485.log 2023-12-04T18:46:01.647 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.89824.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89824.log.gz 2023-12-04T18:46:01.647 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114425.log 2023-12-04T18:46:01.649 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115485.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115485.log.gz 2023-12-04T18:46:01.649 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117532.log 2023-12-04T18:46:01.649 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114425.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114425.log.gz 2023-12-04T18:46:01.652 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89653.log 2023-12-04T18:46:01.652 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.117532.log: 70.2% -- replaced with /var/log/ceph/cluster1-client.admin.117532.log.gz 2023-12-04T18:46:01.652 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99563.log 2023-12-04T18:46:01.652 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.89653.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89653.log.gz 2023-12-04T18:46:01.653 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92069.log 2023-12-04T18:46:01.656 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.99563.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99563.log.gz 2023-12-04T18:46:01.656 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96117.log 2023-12-04T18:46:01.656 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.92069.log: 59.1% -- replaced with /var/log/ceph/cluster2-client.mirror.92069.log.gz 2023-12-04T18:46:01.656 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104761.log 2023-12-04T18:46:01.657 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96117.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.96117.log.gz 2023-12-04T18:46:01.657 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92034.log 2023-12-04T18:46:01.657 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.104761.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.104761.log.gz 2023-12-04T18:46:01.659 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113098.log 2023-12-04T18:46:01.659 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.92034.log: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92034.log.gz 2023-12-04T18:46:01.659 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118352.log 2023-12-04T18:46:01.659 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113098.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113098.log.gz 2023-12-04T18:46:01.661 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96234.log 2023-12-04T18:46:01.661 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.118352.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.118352.log.gz 2023-12-04T18:46:01.661 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110198.log 2023-12-04T18:46:01.667 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.96234.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96234.log.gz 2023-12-04T18:46:01.667 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103097.log 2023-12-04T18:46:01.670 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110198.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110198.log.gz 2023-12-04T18:46:01.670 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115185.log 2023-12-04T18:46:01.671 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103097.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103097.log.gz 2023-12-04T18:46:01.671 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103889.log 2023-12-04T18:46:01.673 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115185.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115185.log.gz 2023-12-04T18:46:01.673 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117245.log 2023-12-04T18:46:01.674 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103889.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103889.log.gz 2023-12-04T18:46:01.674 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102392.log 2023-12-04T18:46:01.675 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117245.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117245.log.gz 2023-12-04T18:46:01.675 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95230.log 2023-12-04T18:46:01.677 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102392.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102392.log.gz 2023-12-04T18:46:01.677 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118362.log 2023-12-04T18:46:01.683 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95230.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95230.log.gz 2023-12-04T18:46:01.683 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95643.log 2023-12-04T18:46:01.683 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118362.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118362.log.gz 2023-12-04T18:46:01.684 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92145.log 2023-12-04T18:46:01.685 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95643.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95643.log.gz 2023-12-04T18:46:01.687 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85935.log 2023-12-04T18:46:01.687 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.92145.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92145.log.gz 2023-12-04T18:46:01.687 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103455.log 2023-12-04T18:46:01.687 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.85935.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85935.log.gz 2023-12-04T18:46:01.690 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102212.log 2023-12-04T18:46:01.690 INFO:teuthology.orchestra.run.smithi176.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91862.log.gz 2023-12-04T18:46:01.691 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91759.log 2023-12-04T18:46:01.691 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102212.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102212.log.gz 2023-12-04T18:46:01.691 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94640.log 2023-12-04T18:46:01.691 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.91759.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91759.log.gz 2023-12-04T18:46:01.691 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108776.log 2023-12-04T18:46:01.696 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94640.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94640.log.gz 2023-12-04T18:46:01.696 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104390.log 2023-12-04T18:46:01.697 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108776.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108776.log.gz 2023-12-04T18:46:01.698 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93306.log 2023-12-04T18:46:01.698 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104390.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104390.log.gz 2023-12-04T18:46:01.699 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111456.log 2023-12-04T18:46:01.712 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117025.log 2023-12-04T18:46:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111456.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111456.log.gz 2023-12-04T18:46:01.726 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104469.log 2023-12-04T18:46:01.727 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117025.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117025.log.gz 2023-12-04T18:46:01.739 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100999.log.gz 2023-12-04T18:46:01.740 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106674.log 2023-12-04T18:46:01.740 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104469.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104469.log.gz 2023-12-04T18:46:01.741 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.91283.log 2023-12-04T18:46:01.742 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106674.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106674.log.gz 2023-12-04T18:46:01.743 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91811.log 2023-12-04T18:46:01.754 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.2.91283.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.91283.log.gz 2023-12-04T18:46:01.755 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114485.log 2023-12-04T18:46:01.756 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.91811.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.91811.log.gz 2023-12-04T18:46:01.756 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104491.log 2023-12-04T18:46:01.757 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114485.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114485.log.gz 2023-12-04T18:46:01.757 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109808.log 2023-12-04T18:46:01.770 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118260.log 2023-12-04T18:46:01.771 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109808.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109808.log.gz 2023-12-04T18:46:01.784 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111097.log 2023-12-04T18:46:01.784 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118260.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118260.log.gz 2023-12-04T18:46:01.785 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97524.log 2023-12-04T18:46:01.785 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111097.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111097.log.gz 2023-12-04T18:46:01.788 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94039.log 2023-12-04T18:46:01.789 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.97524.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.97524.log.gz 2023-12-04T18:46:01.797 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103060.log.gz 2023-12-04T18:46:01.803 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117045.log 2023-12-04T18:46:01.804 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94039.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94039.log.gz 2023-12-04T18:46:01.805 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102750.log 2023-12-04T18:46:01.806 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117045.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117045.log.gz 2023-12-04T18:46:01.806 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116405.log 2023-12-04T18:46:01.807 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102750.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102750.log.gz 2023-12-04T18:46:01.808 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110314.log 2023-12-04T18:46:01.809 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116405.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116405.log.gz 2023-12-04T18:46:01.809 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93924.log 2023-12-04T18:46:01.810 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110314.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110314.log.gz 2023-12-04T18:46:01.810 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105077.log 2023-12-04T18:46:01.811 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93924.log: 74.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93924.log.gz 2023-12-04T18:46:01.812 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111890.log 2023-12-04T18:46:01.812 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105077.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.105077.log.gz 2023-12-04T18:46:01.813 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104617.log 2023-12-04T18:46:01.814 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111890.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111890.log.gz 2023-12-04T18:46:01.814 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108933.log 2023-12-04T18:46:01.815 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104617.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.104617.log.gz 2023-12-04T18:46:01.815 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97500.log 2023-12-04T18:46:01.818 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108933.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.108933.log.gz 2023-12-04T18:46:01.818 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96573.log 2023-12-04T18:46:01.831 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.97500.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95161.log 2023-12-04T18:46:01.832 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.96573.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96573.log.gz 2023-12-04T18:46:01.845 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116025.log 2023-12-04T18:46:01.846 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95161.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.95161.log.gz 2023-12-04T18:46:01.859 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101526.log 2023-12-04T18:46:01.860 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116025.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116025.log.gz 2023-12-04T18:46:01.868 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117305.log 2023-12-04T18:46:01.869 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101526.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101526.log.gz 2023-12-04T18:46:01.877 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94200.log 2023-12-04T18:46:01.877 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117305.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.117305.log.gz 2023-12-04T18:46:01.886 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95749.log 2023-12-04T18:46:01.891 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94200.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94200.log.gz 2023-12-04T18:46:01.895 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114445.log 2023-12-04T18:46:01.896 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95749.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95749.log.gz 2023-12-04T18:46:01.901 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94704.log.gz 2023-12-04T18:46:01.901 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107316.log 2023-12-04T18:46:01.902 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114445.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114445.log.gz 2023-12-04T18:46:01.902 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99457.log 2023-12-04T18:46:01.903 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107316.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107316.log.gz 2023-12-04T18:46:01.904 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117957.log 2023-12-04T18:46:01.905 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99457.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99457.log.gz 2023-12-04T18:46:01.905 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102888.log 2023-12-04T18:46:01.906 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.117957.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.117957.log.gz 2023-12-04T18:46:01.907 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94331.log 2023-12-04T18:46:01.926 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102888.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99257.log 2023-12-04T18:46:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.94331.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.94331.log.gz 2023-12-04T18:46:01.935 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103831.log.gz 2023-12-04T18:46:01.935 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97974.log 2023-12-04T18:46:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99257.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.99257.log.gz 2023-12-04T18:46:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112257.log 2023-12-04T18:46:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.97974.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.97974.log.gz 2023-12-04T18:46:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.93924.log 2023-12-04T18:46:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112257.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112257.log.gz 2023-12-04T18:46:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101078.log 2023-12-04T18:46:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.0.93924.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.93924.log.gz 2023-12-04T18:46:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112144.log 2023-12-04T18:46:01.959 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99745.log 2023-12-04T18:46:01.960 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112144.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112144.log.gz 2023-12-04T18:46:01.976 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113988.log 2023-12-04T18:46:01.983 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99745.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104491.log.gz 2023-12-04T18:46:01.988 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108726.log 2023-12-04T18:46:01.989 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113988.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113988.log.gz 2023-12-04T18:46:02.003 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116585.log 2023-12-04T18:46:02.003 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.108726.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108726.log.gz 2023-12-04T18:46:02.026 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112185.log 2023-12-04T18:46:02.027 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116585.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116585.log.gz 2023-12-04T18:46:02.036 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98323.log 2023-12-04T18:46:02.038 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112185.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112185.log.gz 2023-12-04T18:46:02.054 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98604.log 2023-12-04T18:46:02.059 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98323.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98323.log.gz 2023-12-04T18:46:02.074 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115685.log 2023-12-04T18:46:02.079 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98604.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98604.log.gz 2023-12-04T18:46:02.084 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106147.log 2023-12-04T18:46:02.084 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115685.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115685.log.gz 2023-12-04T18:46:02.093 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115625.log 2023-12-04T18:46:02.093 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106147.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106147.log.gz 2023-12-04T18:46:02.102 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98905.log 2023-12-04T18:46:02.102 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115625.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115625.log.gz 2023-12-04T18:46:02.116 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97500.log.gz 2023-12-04T18:46:02.117 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111661.log 2023-12-04T18:46:02.123 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.98905.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98905.log.gz 2023-12-04T18:46:02.123 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.91106.log 2023-12-04T18:46:02.124 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111661.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111661.log.gz 2023-12-04T18:46:02.125 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90409.log 2023-12-04T18:46:02.128 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.1.91106.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.91106.log.gz 2023-12-04T18:46:02.128 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106581.log 2023-12-04T18:46:02.129 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90409.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90409.log.gz 2023-12-04T18:46:02.129 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95000.log 2023-12-04T18:46:02.131 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106581.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106581.log.gz 2023-12-04T18:46:02.132 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108639.log 2023-12-04T18:46:02.132 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95000.log: 73.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95000.log.gz 2023-12-04T18:46:02.132 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94361.log 2023-12-04T18:46:02.133 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108639.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108639.log.gz 2023-12-04T18:46:02.133 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90410.log 2023-12-04T18:46:02.135 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94361.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.94361.log.gz 2023-12-04T18:46:02.135 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.96682.log 2023-12-04T18:46:02.136 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90410.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90410.log.gz 2023-12-04T18:46:02.136 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.97121.log 2023-12-04T18:46:02.143 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.2.96682.log: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.96682.log.gz 2023-12-04T18:46:02.144 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117643.log 2023-12-04T18:46:02.149 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.2.97121.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.2.97121.log.gz 2023-12-04T18:46:02.149 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91090.log 2023-12-04T18:46:02.149 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.117643.log: 50.4% -- replaced with /var/log/ceph/cluster1-client.admin.117643.log.gz 2023-12-04T18:46:02.151 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117145.log 2023-12-04T18:46:02.151 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.91090.log: 81.1% -- replaced with /var/log/ceph/cluster2-client.mirror.91090.log.gz 2023-12-04T18:46:02.151 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107405.log 2023-12-04T18:46:02.152 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117145.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117145.log.gz 2023-12-04T18:46:02.152 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116265.log 2023-12-04T18:46:02.155 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107405.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107405.log.gz 2023-12-04T18:46:02.155 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91828.log 2023-12-04T18:46:02.156 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116265.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.116265.log.gz 2023-12-04T18:46:02.156 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90748.log 2023-12-04T18:46:02.158 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.91828.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.91828.log.gz 2023-12-04T18:46:02.159 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.107465.log 2023-12-04T18:46:02.159 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.90748.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90748.log.gz 2023-12-04T18:46:02.159 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113901.log 2023-12-04T18:46:02.163 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.2.107465.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103455.log.gz 2023-12-04T18:46:02.163 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103317.log 2023-12-04T18:46:02.164 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113901.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113901.log.gz 2023-12-04T18:46:02.165 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114058.log 2023-12-04T18:46:02.166 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103317.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103317.log.gz 2023-12-04T18:46:02.166 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117772.log 2023-12-04T18:46:02.167 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.114058.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114058.log.gz 2023-12-04T18:46:02.168 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104528.log 2023-12-04T18:46:02.168 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117772.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.117772.log.gz 2023-12-04T18:46:02.170 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101351.log 2023-12-04T18:46:02.170 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104528.log: 74.0% -- replaced with /var/log/ceph/cluster2-client.mirror.104528.log.gz 2023-12-04T18:46:02.170 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110615.log 2023-12-04T18:46:02.170 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101351.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101351.log.gz 2023-12-04T18:46:02.172 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115325.log 2023-12-04T18:46:02.172 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110615.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110615.log.gz 2023-12-04T18:46:02.172 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115585.log 2023-12-04T18:46:02.173 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115325.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115325.log.gz 2023-12-04T18:46:02.173 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115285.log 2023-12-04T18:46:02.175 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115585.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115585.log.gz 2023-12-04T18:46:02.175 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113648.log 2023-12-04T18:46:02.176 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115285.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115285.log.gz 2023-12-04T18:46:02.176 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.104528.log 2023-12-04T18:46:02.177 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113648.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113648.log.gz 2023-12-04T18:46:02.177 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110702.log 2023-12-04T18:46:02.181 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.0.104528.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.104528.log.gz 2023-12-04T18:46:02.183 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101209.log 2023-12-04T18:46:02.183 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110702.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110702.log.gz 2023-12-04T18:46:02.184 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113467.log 2023-12-04T18:46:02.185 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101209.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101209.log.gz 2023-12-04T18:46:02.186 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113530.log 2023-12-04T18:46:02.186 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113467.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113467.log.gz 2023-12-04T18:46:02.187 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95431.log 2023-12-04T18:46:02.188 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.113530.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113530.log.gz 2023-12-04T18:46:02.189 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106882.log 2023-12-04T18:46:02.190 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95431.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.95431.log.gz 2023-12-04T18:46:02.190 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117456.log 2023-12-04T18:46:02.191 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106882.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106882.log.gz 2023-12-04T18:46:02.191 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94861.log 2023-12-04T18:46:02.192 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117456.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117456.log.gz 2023-12-04T18:46:02.193 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96277.log 2023-12-04T18:46:02.194 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.94861.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.94861.log.gz 2023-12-04T18:46:02.194 INFO:teuthology.orchestra.run.smithi176.stderr:gzip 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.107465.log.gz 2023-12-04T18:46:02.195 INFO:teuthology.orchestra.run.smithi176.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107720.log 2023-12-04T18:46:02.195 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102888.log.gz 2023-12-04T18:46:02.195 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93784.log 2023-12-04T18:46:02.195 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.96277.log: /var/log/ceph/cluster2-client.mirror.107720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94220.log 2023-12-04T18:46:02.196 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93784.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93784.log.gz 2023-12-04T18:46:02.196 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106051.log 2023-12-04T18:46:02.196 INFO:teuthology.orchestra.run.smithi176.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107720.log.gz 2023-12-04T18:46:02.199 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.94220.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96277.log.gz 2023-12-04T18:46:02.200 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91606.log 2023-12-04T18:46:02.201 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106947.log 2023-12-04T18:46:02.201 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.91606.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106051.log.gz 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.91606.log.gz 2023-12-04T18:46:02.201 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:02.201 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116125.log 2023-12-04T18:46:02.203 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106947.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91982.log 2023-12-04T18:46:02.204 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116125.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116125.log.gz 2023-12-04T18:46:02.204 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.94220.log.gz 2023-12-04T18:46:02.204 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97554.log 2023-12-04T18:46:02.204 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106031.log 2023-12-04T18:46:02.204 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.91982.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106947.log.gz 2023-12-04T18:46:02.204 INFO:teuthology.orchestra.run.smithi176.stderr: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.91982.log.gz 2023-12-04T18:46:02.204 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.97554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94436.log 2023-12-04T18:46:02.204 INFO:teuthology.orchestra.run.smithi176.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97554.log.gz 2023-12-04T18:46:02.205 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102074.log 2023-12-04T18:46:02.216 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.106031.log: /var/log/ceph/cluster2-client.mirror.94436.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.106031.log.gz 2023-12-04T18:46:02.216 INFO:teuthology.orchestra.run.smithi176.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94436.log.gz 2023-12-04T18:46:02.216 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114845.log 2023-12-04T18:46:02.217 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105812.log 2023-12-04T18:46:02.217 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102074.log: /var/log/ceph/cluster1-client.mirror.114845.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116685.log 2023-12-04T18:46:02.217 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105812.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114845.log.gz 2023-12-04T18:46:02.217 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102074.log.gz 2023-12-04T18:46:02.217 INFO:teuthology.orchestra.run.smithi176.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.105812.log.gz 2023-12-04T18:46:02.217 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116825.log 2023-12-04T18:46:02.217 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111073.log 2023-12-04T18:46:02.217 INFO:teuthology.orchestra.run.smithi176.stderr: 87.3% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.93306.log.gz 2023-12-04T18:46:02.217 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.116685.log.gz 2023-12-04T18:46:02.217 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95273.log 2023-12-04T18:46:02.218 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116825.log.gz 2023-12-04T18:46:02.218 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100086.log 2023-12-04T18:46:02.218 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.111073.log: /var/log/ceph/cluster2-client.mirror.95273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95895.log 2023-12-04T18:46:02.218 INFO:teuthology.orchestra.run.smithi176.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111073.log.gz 2023-12-04T18:46:02.218 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115405.log 2023-12-04T18:46:02.218 INFO:teuthology.orchestra.run.smithi176.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100086.log.gz 2023-12-04T18:46:02.218 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109714.log 2023-12-04T18:46:02.218 INFO:teuthology.orchestra.run.smithi176.stderr: 87.8%/var/log/ceph/cluster1-client.mirror.115405.log: -- replaced with /var/log/ceph/cluster1-client.mirror.95895.log.gz 2023-12-04T18:46:02.219 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101746.log 2023-12-04T18:46:02.219 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115405.log.gz 2023-12-04T18:46:02.219 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104634.log 2023-12-04T18:46:02.219 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109714.log: /var/log/ceph/cluster2-client.mirror.101746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97820.log 2023-12-04T18:46:02.219 INFO:teuthology.orchestra.run.smithi176.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101746.log.gz/var/log/ceph/cluster2-client.mirror.104634.log: 2023-12-04T18:46:02.219 INFO:teuthology.orchestra.run.smithi176.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109714.log.gz 2023-12-04T18:46:02.219 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97232.log 2023-12-04T18:46:02.219 INFO:teuthology.orchestra.run.smithi176.stderr: 88.7%/var/log/ceph/cluster1-client.mirror.97820.log: -- replaced with /var/log/ceph/cluster2-client.mirror.104634.log.gz 2023-12-04T18:46:02.219 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102096.log 2023-12-04T18:46:02.220 INFO:teuthology.orchestra.run.smithi176.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.97820.log.gz 2023-12-04T18:46:02.220 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114385.log 2023-12-04T18:46:02.220 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.97232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102671.log 2023-12-04T18:46:02.220 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102096.log: /var/log/ceph/cluster1-client.mirror.114385.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97232.log.gz 2023-12-04T18:46:02.220 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114385.log.gz 2023-12-04T18:46:02.220 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93081.log 2023-12-04T18:46:02.220 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102671.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101805.log 2023-12-04T18:46:02.220 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95273.log.gz 2023-12-04T18:46:02.220 INFO:teuthology.orchestra.run.smithi176.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102671.log.gz 2023-12-04T18:46:02.221 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96917.log 2023-12-04T18:46:02.221 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93197.log 2023-12-04T18:46:02.221 INFO:teuthology.orchestra.run.smithi176.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93081.log.gz 2023-12-04T18:46:02.221 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101192.log 2023-12-04T18:46:02.221 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93197.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93197.log.gz 2023-12-04T18:46:02.221 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112165.log 2023-12-04T18:46:02.221 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101192.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101192.log.gz 2023-12-04T18:46:02.221 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109764.log 2023-12-04T18:46:02.228 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112165.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112165.log.gz 2023-12-04T18:46:02.228 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116445.log 2023-12-04T18:46:02.228 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109764.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109764.log.gz 2023-12-04T18:46:02.229 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117968.log 2023-12-04T18:46:02.229 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116445.log: 87.3% 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96917.log.gz 2023-12-04T18:46:02.229 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.116445.log.gz 2023-12-04T18:46:02.229 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98366.log 2023-12-04T18:46:02.229 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.117968.log: 8.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114345.log 2023-12-04T18:46:02.229 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.117968.log.gz 2023-12-04T18:46:02.229 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105282.log 2023-12-04T18:46:02.229 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114345.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114345.log.gz 2023-12-04T18:46:02.230 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118311.log 2023-12-04T18:46:02.230 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.105282.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105282.log.gz 2023-12-04T18:46:02.230 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97121.log 2023-12-04T18:46:02.230 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118311.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118311.log.gz 2023-12-04T18:46:02.230 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98237.log 2023-12-04T18:46:02.230 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.97121.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.97121.log.gz 2023-12-04T18:46:02.230 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118268.log 2023-12-04T18:46:02.230 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98237.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98366.log.gz 2023-12-04T18:46:02.233 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96876.log 2023-12-04T18:46:02.234 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118268.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118268.log.gz 2023-12-04T18:46:02.234 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.91239.log 2023-12-04T18:46:02.234 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.96876.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98237.log.gz 2023-12-04T18:46:02.234 INFO:teuthology.orchestra.run.smithi176.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96876.log.gz 2023-12-04T18:46:02.234 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95571.log 2023-12-04T18:46:02.234 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.1.91239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113508.log 2023-12-04T18:46:02.234 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95571.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.95571.log.gz 2023-12-04T18:46:02.234 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98452.log 2023-12-04T18:46:02.235 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113508.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113508.log.gz 2023-12-04T18:46:02.235 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101489.log 2023-12-04T18:46:02.238 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98452.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.91239.log.gz 2023-12-04T18:46:02.238 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109350.log 2023-12-04T18:46:02.241 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101489.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98452.log.gz 2023-12-04T18:46:02.241 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117185.log 2023-12-04T18:46:02.241 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109350.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109350.log.gz 2023-12-04T18:46:02.242 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95595.log 2023-12-04T18:46:02.242 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117185.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.117185.log.gz 2023-12-04T18:46:02.243 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112869.log 2023-12-04T18:46:02.243 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.95595.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.95595.log.gz 2023-12-04T18:46:02.244 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95865.log 2023-12-04T18:46:02.245 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112869.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112869.log.gz 2023-12-04T18:46:02.245 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103238.log 2023-12-04T18:46:02.246 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.95865.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.95865.log.gz 2023-12-04T18:46:02.246 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97028.log 2023-12-04T18:46:02.247 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103238.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103238.log.gz 2023-12-04T18:46:02.247 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99605.log 2023-12-04T18:46:02.248 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.97028.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.97028.log.gz 2023-12-04T18:46:02.248 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106171.log 2023-12-04T18:46:02.249 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99605.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.99605.log.gz 2023-12-04T18:46:02.249 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114925.log 2023-12-04T18:46:02.250 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106171.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.106171.log.gz 2023-12-04T18:46:02.250 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108451.log 2023-12-04T18:46:02.251 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114925.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114925.log.gz 2023-12-04T18:46:02.252 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115765.log 2023-12-04T18:46:02.253 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108451.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108451.log.gz 2023-12-04T18:46:02.254 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95707.log 2023-12-04T18:46:02.254 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115765.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115765.log.gz 2023-12-04T18:46:02.255 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92088.log 2023-12-04T18:46:02.261 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.95707.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95707.log.gz 2023-12-04T18:46:02.261 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89866.log 2023-12-04T18:46:02.261 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.92088.log: 56.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92088.log.gz 2023-12-04T18:46:02.262 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94456.log 2023-12-04T18:46:02.262 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.89866.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89866.log.gz 2023-12-04T18:46:02.263 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101588.log 2023-12-04T18:46:02.263 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94456.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.94456.log.gz 2023-12-04T18:46:02.263 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114038.log 2023-12-04T18:46:02.264 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101588.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101588.log.gz 2023-12-04T18:46:02.264 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101726.log 2023-12-04T18:46:02.265 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.114038.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.114038.log.gz 2023-12-04T18:46:02.266 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94387.log 2023-12-04T18:46:02.275 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100225.log 2023-12-04T18:46:02.282 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94387.log: 89.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115305.log 2023-12-04T18:46:02.282 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.94387.log.gz 2023-12-04T18:46:02.295 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112566.log 2023-12-04T18:46:02.296 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.115305.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115305.log.gz 2023-12-04T18:46:02.309 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104274.log 2023-12-04T18:46:02.310 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112566.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112566.log.gz 2023-12-04T18:46:02.322 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107904.log 2023-12-04T18:46:02.323 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104274.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104274.log.gz 2023-12-04T18:46:02.332 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98193.log 2023-12-04T18:46:02.332 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107904.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107904.log.gz 2023-12-04T18:46:02.347 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101078.log.gz 2023-12-04T18:46:02.353 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102688.log 2023-12-04T18:46:02.373 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98193.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98193.log.gz 2023-12-04T18:46:02.378 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114290.log 2023-12-04T18:46:02.379 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102688.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102688.log.gz 2023-12-04T18:46:02.391 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92162.log 2023-12-04T18:46:02.392 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.114290.log: 55.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114290.log.gz 2023-12-04T18:46:02.411 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113279.log 2023-12-04T18:46:02.412 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.92162.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92162.log.gz 2023-12-04T18:46:02.420 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105646.log 2023-12-04T18:46:02.423 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.113279.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99745.log.gz 2023-12-04T18:46:02.424 INFO:teuthology.orchestra.run.smithi176.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113279.log.gz 2023-12-04T18:46:02.425 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116045.log 2023-12-04T18:46:02.426 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.105646.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105646.log.gz 2023-12-04T18:46:02.427 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103117.log 2023-12-04T18:46:02.427 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116045.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116045.log.gz 2023-12-04T18:46:02.428 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102175.log 2023-12-04T18:46:02.429 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103117.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103117.log.gz 2023-12-04T18:46:02.429 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108109.log 2023-12-04T18:46:02.438 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101271.log 2023-12-04T18:46:02.439 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.108109.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108109.log.gz 2023-12-04T18:46:02.448 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105981.log 2023-12-04T18:46:02.448 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101271.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101271.log.gz 2023-12-04T18:46:02.450 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101726.log.gz 2023-12-04T18:46:02.461 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93831.log 2023-12-04T18:46:02.463 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.105981.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105981.log.gz 2023-12-04T18:46:02.463 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107111.log 2023-12-04T18:46:02.464 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.93831.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.93831.log.gz 2023-12-04T18:46:02.465 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118060.log 2023-12-04T18:46:02.466 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107111.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107111.log.gz 2023-12-04T18:46:02.466 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95401.log 2023-12-04T18:46:02.466 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.118060.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118060.log.gz 2023-12-04T18:46:02.468 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110892.log 2023-12-04T18:46:02.468 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.95401.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.95401.log.gz 2023-12-04T18:46:02.469 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90031.log 2023-12-04T18:46:02.470 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110892.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110892.log.gz 2023-12-04T18:46:02.470 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109672.log 2023-12-04T18:46:02.470 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.90031.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90031.log.gz 2023-12-04T18:46:02.471 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118429.log 2023-12-04T18:46:02.473 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.109672.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109672.log.gz 2023-12-04T18:46:02.473 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103497.log 2023-12-04T18:46:02.475 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.118429.log: 8.2% -- replaced with /var/log/ceph/cluster1-client.admin.118429.log.gz 2023-12-04T18:46:02.475 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93678.log 2023-12-04T18:46:02.476 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103497.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103497.log.gz 2023-12-04T18:46:02.476 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101763.log 2023-12-04T18:46:02.478 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93678.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.93678.log.gz 2023-12-04T18:46:02.478 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98624.log 2023-12-04T18:46:02.478 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.101763.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101763.log.gz 2023-12-04T18:46:02.479 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114198.log 2023-12-04T18:46:02.485 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98624.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98624.log.gz 2023-12-04T18:46:02.487 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91541.log 2023-12-04T18:46:02.487 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.114198.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.114198.log.gz 2023-12-04T18:46:02.487 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.99997.log 2023-12-04T18:46:02.487 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.91541.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91541.log.gz 2023-12-04T18:46:02.488 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2023-12-04T18:46:02.492 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.1.99997.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.99997.log.gz 2023-12-04T18:46:02.492 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106629.log 2023-12-04T18:46:02.505 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99197.log 2023-12-04T18:46:02.506 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106629.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106629.log.gz 2023-12-04T18:46:02.519 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97393.log 2023-12-04T18:46:02.520 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99197.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99197.log.gz 2023-12-04T18:46:02.521 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93699.log 2023-12-04T18:46:02.528 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.97393.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.91862.log 2023-12-04T18:46:02.539 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.93699.log: 93.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.99108.log 2023-12-04T18:46:02.539 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.93699.log.gz 2023-12-04T18:46:02.541 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.3.91862.log: -- replaced with /var/log/ceph/cluster2-client.mirror.97393.log.gz 2023-12-04T18:46:02.552 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101805.log.gz 2023-12-04T18:46:02.554 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102434.log 2023-12-04T18:46:02.558 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.0.99108.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.99108.log.gz 2023-12-04T18:46:02.566 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97324.log 2023-12-04T18:46:02.567 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102434.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102434.log.gz 2023-12-04T18:46:02.577 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110913.log 2023-12-04T18:46:02.577 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.97324.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97324.log.gz 2023-12-04T18:46:02.588 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96593.log 2023-12-04T18:46:02.590 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.110913.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110913.log.gz 2023-12-04T18:46:02.591 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100225.log.gz 2023-12-04T18:46:02.591 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100383.log 2023-12-04T18:46:02.592 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.96593.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.96593.log.gz 2023-12-04T18:46:02.592 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112479.log 2023-12-04T18:46:02.605 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85855.log 2023-12-04T18:46:02.606 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112479.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112479.log.gz 2023-12-04T18:46:02.620 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112636.log 2023-12-04T18:46:02.620 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.85855.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85855.log.gz 2023-12-04T18:46:02.621 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116305.log 2023-12-04T18:46:02.621 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.112636.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112636.log.gz 2023-12-04T18:46:02.634 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117085.log 2023-12-04T18:46:02.635 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.116305.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116305.log.gz 2023-12-04T18:46:02.641 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100708.log 2023-12-04T18:46:02.642 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.117085.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102096.log.gz 2023-12-04T18:46:02.643 INFO:teuthology.orchestra.run.smithi176.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.117085.log.gz 2023-12-04T18:46:02.644 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103948.log 2023-12-04T18:46:02.659 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.100708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98084.log 2023-12-04T18:46:02.659 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103948.log: 78.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103948.log.gz 2023-12-04T18:46:02.669 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.100745.log 2023-12-04T18:46:02.669 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98084.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98084.log.gz 2023-12-04T18:46:02.681 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104311.log 2023-12-04T18:46:02.686 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.0.100745.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.100745.log.gz 2023-12-04T18:46:02.688 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101489.log.gz 2023-12-04T18:46:02.696 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99723.log 2023-12-04T18:46:02.697 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.104311.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104311.log.gz 2023-12-04T18:46:02.698 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106810.log 2023-12-04T18:46:02.699 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.99723.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99723.log.gz 2023-12-04T18:46:02.699 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97348.log 2023-12-04T18:46:02.701 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.106810.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106810.log.gz 2023-12-04T18:46:02.701 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94059.log 2023-12-04T18:46:02.702 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.97348.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97348.log.gz 2023-12-04T18:46:02.702 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109945.log 2023-12-04T18:46:02.703 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.94059.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94059.log.gz 2023-12-04T18:46:02.703 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112985.log 2023-12-04T18:46:02.706 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.109945.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109945.log.gz 2023-12-04T18:46:02.706 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110451.log 2023-12-04T18:46:02.707 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.112985.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112985.log.gz 2023-12-04T18:46:02.708 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92052.log 2023-12-04T18:46:02.710 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110451.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110451.log.gz 2023-12-04T18:46:02.710 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93173.log 2023-12-04T18:46:02.711 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.92052.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92052.log.gz 2023-12-04T18:46:02.711 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89950.log 2023-12-04T18:46:02.713 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.93173.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93173.log.gz 2023-12-04T18:46:02.713 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107880.log 2023-12-04T18:46:02.713 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.89950.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89950.log.gz 2023-12-04T18:46:02.715 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103869.log 2023-12-04T18:46:02.715 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.107880.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107880.log.gz 2023-12-04T18:46:02.715 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110639.log 2023-12-04T18:46:02.716 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.103869.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103869.log.gz 2023-12-04T18:46:02.716 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.103948.log 2023-12-04T18:46:02.718 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.110639.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110639.log.gz 2023-12-04T18:46:02.719 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98753.log 2023-12-04T18:46:02.731 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.2.103948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102414.log 2023-12-04T18:46:02.738 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.mirror.98753.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93436.log 2023-12-04T18:46:02.744 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.mirror.102414.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98753.log.gz 2023-12-04T18:46:02.744 INFO:teuthology.orchestra.run.smithi176.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.2.103948.log.gz 2023-12-04T18:46:02.757 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118155.log 2023-12-04T18:46:02.758 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster1-client.admin.93436.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.93436.log.gz 2023-12-04T18:46:02.759 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cluster2-client.admin.118155.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.118155.log.gz 2023-12-04T18:46:02.772 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100383.log.gz 2023-12-04T18:46:02.828 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102175.log.gz 2023-12-04T18:46:02.898 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102414.log.gz 2023-12-04T18:46:02.901 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100708.log.gz 2023-12-04T18:46:04.268 INFO:teuthology.orchestra.run.smithi176.stderr:gzip: /var/log/ceph/cluster1-client.mirror.3.91862.log: file size changed while zipping 2023-12-04T18:46:04.268 INFO:teuthology.orchestra.run.smithi176.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.3.91862.log.gz 2023-12-04T18:46:04.649 INFO:teuthology.orchestra.run.smithi146.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2023-12-04T18:46:05.193 INFO:teuthology.orchestra.run.smithi146.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2023-12-04T18:46:05.661 INFO:teuthology.orchestra.run.smithi146.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2023-12-04T18:46:05.663 INFO:teuthology.orchestra.run.smithi146.stderr: 2023-12-04T18:46:05.663 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m6.395s 2023-12-04T18:46:05.663 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m18.344s 2023-12-04T18:46:05.663 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m0.703s 2023-12-04T18:46:39.368 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2023-12-04T18:46:39.439 INFO:teuthology.orchestra.run.smithi176.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2023-12-04T18:46:51.338 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2023-12-04T18:46:51.340 INFO:teuthology.orchestra.run.smithi176.stderr: 2023-12-04T18:46:51.340 INFO:teuthology.orchestra.run.smithi176.stderr:real 0m52.046s 2023-12-04T18:46:51.340 INFO:teuthology.orchestra.run.smithi176.stderr:user 2m24.366s 2023-12-04T18:46:51.340 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m6.099s 2023-12-04T18:46:51.340 INFO:tasks.ceph:Archiving logs... 2023-12-04T18:46:51.341 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148/remote/smithi146/log 2023-12-04T18:46:51.342 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-04T18:46:53.849 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148/remote/smithi176/log 2023-12-04T18:46:53.850 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-04T18:47:14.198 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2023-12-04T18:47:14.203 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-04T18:47:14.214 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-12-04T18:47:14.222 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-04T18:47:14.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-04T18:47:14.236 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-04T18:47:14.317 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-04T18:47:14.329 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-04T18:47:14.329 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-04T18:47:14.331 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-04T18:47:14.345 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2023-12-04T18:47:14.348 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2023-12-04T18:47:14.350 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-04T18:47:14.350 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2023-12-04T18:47:14.350 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +929us[ +929us] +/- 78ms 2023-12-04T18:47:14.350 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv02.front.sepia.ceph.com 3 10 377 770 -1580us[-1587us] +/- 60ms 2023-12-04T18:47:14.350 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 55 +721us[ +721us] +/- 81ms 2023-12-04T18:47:14.351 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T18:47:14.352 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-04T18:47:14.352 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2023-12-04T18:47:14.352 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +978us[ +978us] +/- 78ms 2023-12-04T18:47:14.352 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv02.front.sepia.ceph.com 3 8 377 188 -1539us[-1537us] +/- 69ms 2023-12-04T18:47:14.353 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 122 +774us[ +774us] +/- 80ms 2023-12-04T18:47:14.353 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T18:47:14.354 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-04T18:47:14.365 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-04T18:47:14.365 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-04T18:47:14.376 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T18:47:14.492 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T18:47:14.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T18:47:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T18:47:14.626 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T18:47:14.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T18:47:14.659 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-04T18:47:14.671 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-04T18:47:14.683 INFO:teuthology.task.internal:Duration was 17390.775465 seconds 2023-12-04T18:47:14.684 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-04T18:47:14.695 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-04T18:47:14.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T18:47:14.698 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T18:47:14.740 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T18:47:14.744 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T18:47:15.084 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-04T18:47:15.084 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2023-12-04T18:47:15.085 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T18:47:15.104 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2023-12-04T18:47:15.104 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T18:47:15.124 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-04T18:47:15.124 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T18:47:15.147 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T18:47:15.194 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-04T18:47:15.194 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T18:47:15.215 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T18:47:15.345 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-04T18:47:15.358 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-04T18:47:15.359 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T18:47:15.385 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T18:47:15.413 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-04T18:47:15.427 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T18:47:15.429 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T18:47:15.452 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2023-12-04T18:47:15.478 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2023-12-04T18:47:15.508 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T18:47:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T18:47:15.536 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T18:47:15.562 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi176.front.sepia.ceph.com, flagging run as failed 2023-12-04T18:47:15.562 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-04T18:47:15.574 INFO:teuthology.task.internal:Transferring archived files... 2023-12-04T18:47:15.575 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148/remote/smithi146 2023-12-04T18:47:15.575 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T18:47:15.611 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148/remote/smithi176 2023-12-04T18:47:15.611 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T18:47:52.701 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-12-04T18:47:52.736 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148/remote/smithi176/coredump/1701700960.96378.core.gz: gzip compressed data, was "1701700960.96378.core", last modified: Mon Dec 4 18:47:15 2023, max compression, original size modulo 2^32 643428352 2023-12-04T18:47:52.736 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-12-04T18:47:55.298 INFO:teuthology.task.internal: core looks like: /tmp/tmpzsiq3f7d: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'rbd-mirror --foreground --cluster cluster2 --id mirror.1', real uid: 1000, effective uid: 1000, real gid: 1301, effective gid: 1301, execfn: '/usr/bin/rbd-mirror', platform: 'x86_64' 2023-12-04T18:47:55.359 INFO:teuthology.task.internal: dump_program: rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-04T18:47:55.360 DEBUG:teuthology.orchestra.run.smithi176:> which 'rbd-mirror --foreground --cluster cluster2 --id mirror.1' 2023-12-04T18:47:55.379 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/which: unrecognized option '--foreground' 2023-12-04T18:47:55.379 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/which: unrecognized option '--cluster' 2023-12-04T18:47:55.380 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/which: unrecognized option '--id' 2023-12-04T18:47:55.380 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/which: no cluster2 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2023-12-04T18:47:55.380 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/which: no mirror.1 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2023-12-04T18:47:55.380 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-12-04T18:47:55.381 INFO:teuthology.orchestra.run.smithi176.stdout:/usr/bin/rbd-mirror 2023-12-04T18:47:55.381 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi176 with status 2: "which 'rbd-mirror --foreground --cluster cluster2 --id mirror.1'" 2023-12-04T18:47:55.394 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-04T18:47:55.409 INFO:teuthology.task.internal:Not uploading archives. 2023-12-04T18:47:55.409 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-04T18:47:55.424 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-04T18:47:55.424 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T18:47:55.430 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T18:47:55.446 INFO:teuthology.orchestra.run.smithi146.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 4 18:47 /home/ubuntu/cephtest 2023-12-04T18:47:55.446 INFO:teuthology.orchestra.run.smithi146.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 4 18:47 /home/ubuntu/cephtest/archive 2023-12-04T18:47:55.446 INFO:teuthology.orchestra.run.smithi146.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 18:47 /home/ubuntu/cephtest/archive/syslog 2023-12-04T18:47:55.446 INFO:teuthology.orchestra.run.smithi146.stdout: 262223 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Dec 4 13:57 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-12-04T18:47:55.447 INFO:teuthology.orchestra.run.smithi146.stdout: 265245 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Dec 4 13:57 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-12-04T18:47:55.447 INFO:teuthology.orchestra.run.smithi146.stdout: 265244 104 -rw-r--r-- 1 ubuntu ubuntu 104714 Dec 4 18:47 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T18:47:55.447 INFO:teuthology.orchestra.run.smithi146.stdout: 402708 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 18:47 /home/ubuntu/cephtest/archive/audit 2023-12-04T18:47:55.447 INFO:teuthology.orchestra.run.smithi146.stdout: 402987 236 -rw------- 1 ubuntu ubuntu 238560 Dec 4 18:47 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-12-04T18:47:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T18:47:55.448 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-04T18:47:55.448 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-12-04T18:47:55.457 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-04T18:47:55.470 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-04T18:47:55.483 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: error during thrashing 2023-12-04T18:47:55.483 INFO:teuthology.nuke:Checking targets against current locks 2023-12-04T18:47:55.490 INFO:teuthology.orchestra.run.smithi176.stdout: 262166 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 4 18:47 /home/ubuntu/cephtest 2023-12-04T18:47:55.490 INFO:teuthology.orchestra.run.smithi176.stdout: 402661 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 4 14:06 /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-04T18:47:55.491 INFO:teuthology.orchestra.run.smithi176.stdout: 402662 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 4 14:07 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-04T18:47:55.491 INFO:teuthology.orchestra.run.smithi176.stdout: 416638 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 14:07 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2023-12-04T18:47:55.491 INFO:teuthology.orchestra.run.smithi176.stdout: 262167 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Dec 4 18:47 /home/ubuntu/cephtest/archive 2023-12-04T18:47:55.491 INFO:teuthology.orchestra.run.smithi176.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 18:47 /home/ubuntu/cephtest/archive/syslog 2023-12-04T18:47:55.491 INFO:teuthology.orchestra.run.smithi176.stdout: 265280 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Dec 4 13:57 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-12-04T18:47:55.491 INFO:teuthology.orchestra.run.smithi176.stdout: 265245 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Dec 4 13:57 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-12-04T18:47:55.491 INFO:teuthology.orchestra.run.smithi176.stdout: 265244 108 -rw-r--r-- 1 ubuntu ubuntu 107063 Dec 4 18:47 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T18:47:55.491 INFO:teuthology.orchestra.run.smithi176.stdout: 402659 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 18:47 /home/ubuntu/cephtest/archive/audit 2023-12-04T18:47:55.492 INFO:teuthology.orchestra.run.smithi176.stdout: 402663 240 -rw------- 1 ubuntu ubuntu 242980 Dec 4 18:47 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-12-04T18:47:55.492 INFO:teuthology.orchestra.run.smithi176.stdout: 262168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 14:42 /home/ubuntu/cephtest/archive/coredump 2023-12-04T18:47:55.492 INFO:teuthology.orchestra.run.smithi176.stdout: 268181 58988 -rw------- 1 ubuntu ubuntu 643428352 Dec 4 14:42 /home/ubuntu/cephtest/archive/coredump/1701700960.96378.core 2023-12-04T18:47:55.492 INFO:teuthology.orchestra.run.smithi176.stdout: 268180 48148 -rw------- 1 ubuntu ubuntu 631492608 Dec 4 14:42 /home/ubuntu/cephtest/archive/coredump/1701700939.103948.core 2023-12-04T18:47:55.493 INFO:teuthology.orchestra.run.smithi176.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-04T18:47:55.512 DEBUG:teuthology.nuke:shortname: smithi146 2023-12-04T18:47:55.513 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T18:47:55.533 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 13:48:04.743361', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCfbmNcyqeHhjNg3SpAV70mufKp07o/mtrEKQB1dLC4Q+M8Bfmnpw6lvzxhJ5bkx+i89G8/Yven4H30c2gnqfP8='} 2023-12-04T18:47:55.537 DEBUG:teuthology.nuke:shortname: smithi176 2023-12-04T18:47:55.537 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T18:47:55.559 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-04_13:47:01-rbd-wip-dis-testing-revert-distro-default-smithi/7477148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 13:48:04.745355', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODo4dFnAcHFCBEVYKDs1fTrIEVFsW1LBi7z472ULX0gW0/1v17likGLMVd2GFpVeSPR9ljaeM34RM4QVXP6ObY='} 2023-12-04T18:47:55.579 INFO:teuthology.orchestra.console.smithi146:Power off 2023-12-04T18:47:55.579 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T18:47:55.601 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2023-12-04T18:47:55.602 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T18:47:55.620 INFO:teuthology.orchestra.console.smithi176:Power off 2023-12-04T18:47:55.620 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T18:47:55.629 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2023-12-04T18:47:55.641 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2023-12-04T18:47:55.642 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T18:47:55.662 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2023-12-04T18:47:59.630 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T18:47:59.664 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T18:47:59.740 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2023-12-04T18:47:59.773 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2023-12-04T18:48:03.742 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T18:48:03.774 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T18:48:03.852 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2023-12-04T18:48:03.883 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2023-12-04T18:48:03.953 INFO:teuthology.orchestra.console.smithi146:Power off completed 2023-12-04T18:48:07.888 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T18:48:07.997 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2023-12-04T18:48:08.098 INFO:teuthology.orchestra.console.smithi176:Power off completed 2023-12-04T18:48:08.204 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} duration: 17390.77546453476 failure_reason: 'Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi176 with status 110: ''mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7c730b2996d0b00d56aea5ca791f90c534a13422 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\''\'' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh''' flavor: default owner: scheduled_dis@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=aa28831dc0f54a009e0a1fb4516b61fb status: fail success: false 2023-12-04T18:48:08.204 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T18:48:08.296 INFO:teuthology.run:FAIL